| All Forums |
Register |
Login |
Search |
Subscriptions |
My Profile |
Inbox |
| Tool Warehouse |
FAQs |
Resources |
Help |
Member List |
Address Book |
Logout |
|
|
How to fix broken pipes?
|
Logged in as: Guest |
| Users viewing this topic: none |
|
Login  |
|
|
 How to fix broken pipes? - Mar. 31, '06, 4:26:17 PM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
How is cat32 supposed to work? I was under the impression that it needed to be applied to non-Interix applications when mixing pipes. A brief test shows this is incorrect; it is the Interix apps that must be filtered?!
$ { cl.exe | cat32 ; ls | cat32 ; } | cat # WORKS
$ { cl.exe ; ls | cat32 ; } | cat # WORKS
$ { cl.exe | cat32 ; ls ; } | cat # BROKEN!
Here is my real problem... I have a makefile that tells 'make' to build a source file by calling 'cl <args>'. If I do it at the command line, no problems. If I pipe it through anything, it apparently EPIPE's (goes *piff*, but once and only once I saw something about 'broken pipe'). 'cl' is a wrapper script that calls another script (with ' | cat32') which munges the arguments and then calls 'cl.exe "${args[@]}" 2>&1 | cat32' (which I now know is pointless). The only other thing I can think of is to change 'make' to add '| cat32' to its exec() call.
Am I overlooking something? I would like to understand just where I need to be placing 'cat32' to avoid this problem, as I am having the same issue in a different (but less "fatal") location as well.
|
|
|
|
 RE: How to fix broken pipes? - Mar. 31, '06, 8:40:36 PM
|
|
|
Rodney
Posts: 3714
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
|
You've misunderstood the explanation.
The cat32 utility is a Win32 program. This allows it to capture the output
from another Win32 program in the Win32 paradigm. Then, being a well written
program, it outputs this information is a more standard/portable way that Unix
programs use. So it is not for Interix programs.
The real problem that cat32 solves is the closing of a pipe. Win32 will
close a pipe at an unexpected time resulting in EPIPE. You see that you've made
a subshell with various commands. These will not necessarily start and finish
in the order that you think they will due to system scheduling, quantums, etc.
If you read through the cat32 manual page you'll see the explanations of
what works and doesn't. In the above samples that you give, the last one is failing
because the rightmost pipe is getting the close from the cl.exe part so that the
output form the ls is now trying to be written to a closed pipe.
The cat32 at the end of the subshell is holding the Win32 close on
the write end of the pipe off until all of the writes in the subshell finish.
|
|
|
|
|
RE: How to fix broken pipes? - Apr. 3, '06, 10:45:10 AM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
Wait... does this mean that 'make' will ALWAYS fail with EPIPE after exec()ing any Windows program? If so, how am I supposed to fix this; just re-write 'make' to ignore EPIPE? Or can I ensure that the Windows-program output is always passed through an Interix program? I noticed if I use plain-ol' 'cat' instead of 'cat32', that things seem to work the way I'd expected. Is this reliable?
$ { cl.exe | cat ; ls | cat ; } | cat # WORKS
$ { cl.exe ; ls | cat ; } | cat # BROKEN AS EXPECTED
$ { cl.exe | cat ; ls ; } | cat # WORKS!
[EDIT] Nope, that doesn't work. I have the problem mentioned in my previous post; is there a way to fix this that does not involve editing 'make'? I can do that, too, but I'd be worried about missing a 'correct' EPIPE.
< Message edited by woehlkmp -- Apr. 3, '06, 10:50:15 AM >
|
|
|
|
|
RE: How to fix broken pipes? - Apr. 3, '06, 5:02:09 PM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
I don't see anything in /usr/contrib/win32/bin/ that looks like it is doing anything to pipes... just tweaking some environment variables and checking that the application can be run.
Anyway... here is my crazy idea: write a proper pipe sanitizer. This program would be invoked via (e.g.) 'sanitize <prog> [<args>]' and would fork() and exec() '<prog>' after first creating pipes such that the initial process reads and writes to the exec()'d child's stdin/stdout, such that pipe "problems" are not propogated (because the child's stdin/stdout are overlayed new fd's created by pipe()).
And... it works... on Linux.
On Interix, my poll() never returns. Any guesses?
|
|
|
|
|
RE: How to fix broken pipes? - Apr. 3, '06, 5:20:54 PM
|
|
|
Rodney
Posts: 3714
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
|
>I don't see anything in /usr/contrib/win32/bin/ that looks like it is doing anything to pipes... just tweaking some environment >variables and checking that the application can be run.
I'm just suggesting that this method (the script) might be usable to, as you call it, "sanitize" the output.
This "sanitization" would need to be added to the script that you'll invoke.
> On Interix, my poll() never returns. Any guesses?
And it won't unless your call to poll() is for the /proc filesystem.
It was only written for /proc and not for any other part of the system
(to support the gdb debugger and pstat utility).
It was done in Firebrand (Softway days) and the orginal plan (at Softway)
was for poll() to be completely implemented post-Firebrand. That was 1998/99.
Interix went to MS in 1999. Since then many people have asked for poll() to
be completed. You have to use select() for now.
|
|
|
|
 RE: How to fix broken pipes? - Apr. 3, '06, 6:11:09 PM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
Ooookay. poll() not implemented. Nice of Microsoft to leave a big note about that major ommission. And nice of their compiler to mention this. And nice of them to implement it and have it just plain not work (rather than not implementing it, which would have been a LOT better!).
So... rewrote it with select() (which was much more painful), and it still doesn't work! I don't get it; I am not letting the Windows programs TOUCH any of my std* fd's (I'm dup2()ing ALL of them to pipe()s before execvp()ing), so why oh why is it still broken?
Someone excuse me while I go tear my hair out  .
[EDIT] Sorry, didn't see your (Rodney's) post until I'd posted this... I figured it out myself.
quote:
This "sanitization" would need to be added to the script that you'll invoke.
Oh, ok, sorry for the confusion. That's actually what I'm trying to do (if you read my original post, I am already invoking the Windows tools via a wrapper script, which is what 'make' actually calls, so I've been trying to add my sanitizations there). Unfortunately, I can't find a sanitization that works... I thought the pipe() plus exec() would be a sure thing! ...unless it's some particular character causing the problem?
[/EDIT]
< Message edited by woehlkmp -- Apr. 3, '06, 6:13:44 PM >
|
|
|
|
|
RE: How to fix broken pipes? - Apr. 6, '06, 4:51:22 AM
|
|
|
mduft
Posts: 15
Joined: Nov. 16, '05,
Status: offline
|
Hey there!
I had similar problems a while ago... And i have a (ok, it's ugly... ;o) solution:
tail the whole thing:
rm -f somelogfile.log \
touch somelogfile.log \
tail -f somelogfile.log & 2>&1 \
export TAIL_PID=$! \
( #some commands...
) > somelogfile.log 2>&1 | tee -a someotherlogfile.log \ #(or some other command to pipe to...)
sleep 2 \ # give tail some time to output everything (updates every second)
kill ${TAIL_PID} \ # kill the tail
rm -f somelogfile.log
be sure to connect the commands (with "\") so they get run in the same shell, and TAIL_PID won't get lost underway.
I hope this helps... it at least worked for me for a while now... As soon as you redirect output from the while stuff, things begin to work, so it seems to have something todo with stdin/stdout in that case. I also tried writing an own entry point for windows applications (_mainCRTStartup) and not closing all file and pipe handles on exit, but it didn't work...
Regards, Markus
|
|
|
|
|
RE: How to fix broken pipes? - Apr. 6, '06, 12:45:53 PM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
Thanks for sharing! I did finally get my solution working, though; on 32-bit it even works without the fork delay hack.
|
|
|
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
|
 Locked w/ New Messages |
 Locked w/o New Messages |
|
 Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts |
|
|
|
Printable Version
RE: How to fix broken pipes? - 