| All Forums |
Register |
Login |
Search |
Subscriptions |
My Profile |
Inbox |
| Tool Warehouse |
FAQs |
Resources |
Help |
Member List |
Address Book |
Logout |
|
|
RE: Several "common" utils gone in 5.2; 3.5 versions unusable
|
Logged in as: Guest |
| Users viewing this topic: none |
|
Login  |
|
|
 RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 12:21:22 PM
|
|
|
breiter
Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
|
quote:
ORIGINAL: woehlkmp
The problem I was trying to get around is that winpath2unix, by itself, treats '/usr' like '\usr', which results in things like '/dev/fs/C/usr' rather than '/usr' like I wanted! (Similarly, 'unixpath2win C:\\' just doesn't work; again, not what I want.)
I don't follow you. winpath2unix(2)--the function--resolves relative Win32 paths to posix correctly for me and so does winpath2unix(1)--the utility. The same goes for unixpath2win. The path or file isn't required to exist, it just gets translated. So \user (C:\usr) should resolve to /dev/fs/C/usr which probably doesn't exist.
% winpath2unix "C:\"
/dev/fs/C
% unixpath2win /usr
C:\WINDOWS\SUA\usr
The functions aren't supposed to convert "anything" to a valid path. You have to use a valid path of the correct type. If you're trying to maintain portable code, the simplest thing to do is just to add an #ifdef __INTERIX and check incoming paths for '\\' and in that case call winpath2unix and then, having converted the path you can allow the program to execute normally.
Interesting about the hard link. You're right, you can do that in NTFS, too. Check out fsutil. NTFS can have mount points, reparse points (junction), and hard links.
< Message edited by breiter -- Mar. 29, '06, 12:23:59 PM >
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 12:58:56 PM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
quote:
The functions aren't supposed to convert "anything" to a valid path. You have to use a valid path of the correct type.
Why? Cygwin's utilities/API's don't have this limitation; they're "smart" enough to figure out what input you gave them; why can't Interix have this functionality built in? As you already noted, this functionality is important if you're trying to write an app that will run in both worlds, because it won't know what format the path you gave it is in except to examine the input at run-time.
Anyway, what I did is similar to your solution (which decides solely on the basis of whether or not the path contains a '\'), except that I provided source for both directions, and mine won't mess up 'C:/foo'. I did, however, notice that I should probably be converting '\' to '/' before calling 'realpath' (this would be consistent with Cygwin's 'cygpath').
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 1:38:59 PM
|
|
|
Rodney
Posts: 3695
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
|
> FYI. It doesn't seem to matter if something is mixed-mode or not. Any posix binary can be invoked from a CMD console without
> using posix.exe in R2. It wasn't like that before was it?
Yup, it was. All the way back to NT 3.5.
The win32 subsystem is actually hardwired to run posix.exe for you when it detects
that the binary is an Interix/Posix binary.
Posix.exe is a special Win32 program that has a lot of special code in it to
talk to the subsystem manager, the Interix subsystem, other special call etc.
So it's not your average Win32 program.
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 1:47:21 PM
|
|
|
Rodney
Posts: 3695
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
|
The winpath2unix() and unixpath2win() API's are actually a lot smarter.
They also resolve /dev/null, "nul", etc. which are quite special things.
They also test for validity. The man page does have most of this information.
These calls are actually system calls that go down to the kernel to map
what the system knows about the paths. They are not simple for-loop convertors.
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 1:48:51 PM
|
|
|
breiter
Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
|
quote:
ORIGINAL: woehlkmp
quote:
The functions aren't supposed to convert "anything" to a valid path. You have to use a valid path of the correct type.
Why? Cygwin's utilities/API's don't have this limitation; they're "smart" enough to figure out what input you gave them; why can't Interix have this functionality built in?
I'm not really defending it, I'm just quoting the documentation of how it works. There may be a good reason it works the way it does. Whoever came up with this API probably thought about it a lot more than I have.
quote:
man 2 winpath2unix
The winpath2unix(2) call converts a Windows pathname to an equivalent
Subsystem for UNIX-based Applications pathname.
If the Windows pathname is a drive-relative pathname such as "\foo", then
the current working directory is translated to Windows syntax using
unixpath2win(2). If this translation fails with EWINPATH, then
winpath2unix(2) also fails with EWINPATH. Otherwise, the root part of the
Windows translation of the current working directory is extracted, which
will be a string such as "C:\" or "\\host\share", and this string is
prepended to the drive-relative pathname. Then the resultant absolute
Windows pathname is translated.
A Windows pathname such as "C:foo" will be interpreted as if it were "C:
\foo". That's the best we can do, since Subsystem for UNIX-based Application
s processes do not have a
notion of a current working directory for each mounted drive.
The Windows pathname is not required to name an existing file. However, if
possible, the winpath2unix(2) call will find the longest prefix of the
Windows pathname that names an existing file which is accessible to the
effective user, and it will convert that prefix to canonical case. This
conversion is necessary because Windows pathname lookups are case
insensitive, while Subsystem for UNIX-based Applications pathname lookups ar
e case sensitive.
I kind of see your point about the C:/foo case. I thought only 0x5c ('\' or "reverse solidus") was guaranteed to be a path separator character and '/' was generally used for switches. I guess it makes sense to test for ':'. In the absense of ':' and '\', you have a relative path that is posixally correct, so it doesn't need to be converted.
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 2:05:27 PM
|
|
|
breiter
Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
|
quote:
ORIGINAL: Rodney
> FYI. It doesn't seem to matter if something is mixed-mode or not. Any posix binary can be invoked from a CMD console without
> using posix.exe in R2. It wasn't like that before was it?
Yup, it was. All the way back to NT 3.5.
The win32 subsystem is actually hardwired to run posix.exe for you when it detects
that the binary is an Interix/Posix binary.
It seems like it is just cmd.exe that is "hardwired" to load posix.exe. You can't launch posix apps from Explorer without psxrun.exe or posix.exe, even mixed-mode apps. It isn't that they can't run, it's just Explorer has no idea what to do with them because there is no extension on the files that matches up with one of the registered "executable extensions" in %PATHEXT" nor is "no extension" a mimetype. Windows doesn't peek at the magic number inside of the file, it just gives up.
Which probably makes my idea of just having a single version of unix utilities that run under Interix but are transparently invokable from Windows apps with no knowledge of the existence of Interix unworkable. For example, redmon doesn't like being given a path to the Interix version of ghostscript instead of the Win32 version.
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 2:07:04 PM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
quote:
I'm not really defending it, I'm just quoting the documentation of how it works. There may be a good reason it works the way it does.
Right. If you know you have a Windows path, then "/foo" -> "/dev/fs/C/foo" is correct (even though "/foo" is probably malformed to begin with). Having explicit converters is probably a good thing, but the lack of implicit converters is unfortunate... a situation we've now corrected, for anyone that finds this thread  .
quote:
In the absense of ':' and '\', you have a relative path that is posixally correct, so it doesn't need to be converted.
Good point. I was writing mine from the standpoint of 'convert this, and you figure out what to do with it' (and also mine does relative->full conversion), but I think your method is OK if you add checking for 'C:/foo'-type paths. As far as I know, these are generally OK to throw around, although '/foo' probably confuses pure-Windows applications.
|
|
|
|
 RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 5:51:54 PM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
quote:
I wonder to what extent you can just fix the problem of duplicate binary images by hard-linking things<snip>
Ah, I missed that; clever! It won't save you if your Windows app spawns an Interix app expecting to do IPC with it, but for anything else, that should work quite nicely.
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 6:05:29 PM
|
|
|
Rodney
Posts: 3695
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
|
> Example:
> ln /bin/telnet /bin/telnet.exe
>
> Now you can invoke %SFUDIR% or %SUADIR%\bin\telnet.exe by clicking on it in explorer.
We used to do this for ksh and csh so they could be specified in a shortcut
without needing to list POSIX.EXE and it's options during the Softway days (Interix 2.2 & prior,
and Firebrand). MS decided different -- don't know why. It works fine.
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 10, '06, 10:51:32 AM
|
|
|
DrPizza
Posts: 40
Joined: Mar. 27, '03,
Status: offline
|
quote:
ORIGINAL: breiter
You can't use Interix to mount filesystems. It is done via diskmgmt.msc. There are, unfortunately, two kinds of symbolic links: the NTFS junction points (made with linkd.exe) and POSIX symlinks (made with ln). Oh, and then there are .lnk files which Explorer uses.
And in Vista/Longhorn Server, there are native symlinks too.
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/symbolic_links.asp
I believe behind the scenes they're implemented with reparse points.
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 10, '06, 11:01:42 AM
|
|
|
breiter
Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
|
I suppose it is too much to hope that Longhorn/Win32 symlinks are actually the same thing as Longhorn/Interix symlinks?
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 10, '06, 11:03:58 AM
|
|
|
DrPizza
Posts: 40
Joined: Mar. 27, '03,
Status: offline
|
It would certainly make sense if they got rid of the old "system file" style symlinks that we see on SUA/SFU in favour of using the proper symlinks.
I don't know if that's sufficient justification, though....
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 10, '06, 1:25:55 PM
|
|
|
Rodney
Posts: 3695
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
|
Reparse points and symbolic links are different.
Symbolic links can point to files or directories.
Reparse points, if I recall correctly, can't point to files.
What will actually ship in Vista/Longhorn I'd wait (not hold my breath) and see.
Back several years ago when reparse points were on the horizon MS claimed that it
would be the same as symbolic links. It wasn't. Many different people and org's
told them so and MS was "surprised" with this information.
< Message edited by Rodney -- Apr. 10, '06, 1:35:01 PM >
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 10, '06, 8:04:57 PM
|
|
|
DrPizza
Posts: 40
Joined: Mar. 27, '03,
Status: offline
|
quote:
Reparse points and symbolic links are different.
Symbolic links can point to files or directories.
Reparse points, if I recall correctly, can't point to files.
A reparse point can do whatever you want. If you want to write a reparse point mechanism to provide symlink functionality you can do so. Reparse points are just a way of allowing arbitrary "hooks" to be installed into the filesystem; whenever the filesystem driver comes across a reparse point it gets the relevant reparse point handler to deal with the reparse point.
The reparse point just allows the RP driver to store a chunk of arbitrary data in the file (I think limited to something like 64 kibytes, but enough for these purposes). Writing an RP driver that stored a relative or absolute path in its metadata would be quite easy.
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 10, '06, 11:59:35 PM
|
|
|
jerker_back
Posts: 68
Joined: Jul. 7, '05,
From: Sweden
Status: offline
|
NTFS "Hardlinks" - duplicate link to a file and NTFS "Junctions" - duplicate mount of a directory (see Hard Links and Junctions) are Microsoft implementations of "Reparse points". These implementations only slightly made it to the WIN32 userlevel (CreateHardLink in kernel32.dll) and are not used in Windows or any other MS products what I know of. Reparse points are mainly intended for developing File system filter drivers and allmost all info and declarations about reparse points is also in the not freely available ntifs.h (Windows DDK IFS-kit). Understanding the fundamentals of the NTFS filesystem is needed to grasp the somewhat confusing aspect of these things.
A junction is not an object, it is a duplicate entry to data streams. So when you delete a junction you will also delete the data pointed to, leaving the original directory empty. The hardlink is I think, an attempt to somewhat mimic an unix-like link and somehow transform the reparse point to an object. The result is not reliable and the hardlink easyly loose the connection and points to the old data when the original file is changed. It is safe to delete hardlinks though. A hardlink is created with the FILE_FLAG_OPEN_REPARSE_POINT flag. Volume mount points is also related.
I used to use junctions a lot before and to avoid the misstake of deleting I made a simple shell extension:
implement ICopyHook::CopyCallback
test file attributes for directories and FILE_ATTRIBUTE_REPARSE_POINT
in case of copy, delete, move or rename a reparse point (FO_COPY,FO_DELETE,FO_MOVE,FO_RENAME)
unlink the junction, do the thing
This worked perfectly but should really have been implemented by MS if they thought we would use these things.
There was some custom solutions of ln.exe hardlinks (low level) that worked on NT4 but I seem to have lost the source. An excellent source to junctions is available at Code project - MakeLink made by Mikael Nordell. The tclWinFile.c from the TCL project is floating around some open source projects (ie Apache).
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 11, '06, 10:46:27 AM
|
|
|
woehlkmp
Posts: 102
Status: offline
|
> This worked perfectly but should really have been implemented by MS if they thought we would use these things.
My sentiments exactly. I am vaguely aware of the existance of such things, but because they are so poorly exposed, they are of almost zero use.
(Something else is Explorer really needs to have a third color for displaying links of any kind, or at least any that are not well-behaved hardlinks (since in those cases, it doesn't matter as much if you don't realize it's a link)... and a column in Details view to show you where a link is points to.)
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 11, '06, 11:02:48 AM
|
|
|
breiter
Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
|
You can manage reparse points with linkd.exe from Microsoft. I think, though, that there is some thought by the security watchdogs at Microsoft (no jokes, please) that they are a "bad idea" from a security perspective. Mike Howard and David LeBlanc discuss a class of canonicalization attacks that are used against UNIX-like systems in Writing Secure Code 2nd edition that Windows has been largely immune to precisely because symlinks aren't really used. So on the one hand they have to have the functionality built into NTFS in order to satisfy certain customers that demand it but on the other hand they don't think it is that great of an idea. Perhaps that explains a little of the schizophrenia we are observing.
Also there is this Explorer add-in to put an overlay on junction points:
http://www.paraesthesia.com/blog/comments.php?id=801_0_1_0_C
And there is this other one that extends the overlay code to include context links to create the junctions and hard links:
http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html
I haven't used either one, though.
< Message edited by breiter -- Apr. 12, '06, 9:27:30 AM >
|
|
|
|
|
RE: Several "common" utils gone in 5.2; 3.5 v... - Apr. 12, '06, 8:57:42 AM
|
|
|
DrPizza
Posts: 40
Joined: Mar. 27, '03,
Status: offline
|
quote:
NTFS "Hardlinks" - duplicate link to a file and NTFS "Junctions" - duplicate mount of a directory (see Hard Links and Junctions) are Microsoft implementations of "Reparse points".
No they aren't. Hard links are hard links--that is, named references to the same reference-counted file. They operate exactly in the same way as Unix hard links. Specifically, hard linked files get muliple $FILE_NAME attributes http://www.linux-ntfs.org/content/view/104/43/#attribute_file_name .
Junctions are a kind of reparse point. Reparse points have different metadata (stored in a $REPARSE_POINT attribute http://www.linux-ntfs.org/content/view/104/43/#attribute_reparse_point ). Junctions have the reparse tag 0x88000003--they're treated as mountpoints. Junctions are quite dangerous because their semantics are not symlink-style semantics; if you delete the junction, you delete the contents of the mountpoint too. This can easily cause data loss if you are unaware that something is a junction.
|
|
|
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
|
 Locked w/ New Messages |
 Locked w/o New Messages |
|
 Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts |
|
|
|
Printable Version
. You might be able to get away with writing 'dumb' wrapper programs (argv[0] might even let you write one and make copies of it) to invoke the Posix counterparts, but you're still stuck with two files, and unfortunately, some programs probably expect to do IPC with certain apps.