Free Downloads, Community Forum,
FAQs and Developer Resources


Make /Tools Your Home | Link to us

Today's posts | Posts since last visit | Most Active Topics

All Forums Register Login Search Subscriptions My Profile Inbox
Tool Warehouse FAQs Resources Help Member List Address Book Logout

Several "common" utils gone in 5.2; 3.5 versions unusable

 
Logged in as: Guest
Users viewing this topic: none
  Printable Version
All Forums >> [SFU / Interix / SUA Technology] >> Windows Server 2003 R2 SUA >> Several "common" utils gone in 5.2; 3.5 versions unusable Page: [1] 2 3 4   next >   >>
Login
Message << Older Topic   Newer Topic >>
Several "common" utils gone in 5.2; 3.5 versi... - Mar. 22, '06, 10:53:48 AM   
breiter

 

Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
I hate having a giant directory tree in my command prompt in any shell. So when I get lost, I'm accustomed to using pwd. Yesterday, I realized that pwd for the win32 command shell was gone after upgrading to Interix 5.2.

cmd> pwd
'pwd' is not recognized as an internal or external command,
operable program or batch file.

At first, I thought my %PATH% environmental variable was messed up, but it isn't. Interix 5.2 "utilities" pack ships with far fewer win32 binaries in it. Here's the list of win32 binaries that exist in the Interix 3.5 "common" suite but not in 5.2:

cron.exe
crontab.exe
iconv.exe
kill.exe
nice.exe
od.exe
paste.exe
printenv.exe
printf.exe
ps.exe
pwd.exe
renice.exe
rshpswd.exe
rshsvc.exe
sdiff.exe
sleep.exe
split.exe
string.exe
su.exe
top.exe
tr.exe
unix2dos.exe
uudecode.exe
uuencode.exe
wait.exe
which.exe
xargs.exe

I don't miss win32 cron or rsh, but some of the others are darn useful. The extra-special part is that if I try using the old binaries on an R2 box, a dialog box pops:

---------------------------
Microsoft Windows Services for UNIX
---------------------------
The license for this pre-release or evaluation version of Microsoft Windows Services for UNIX has expired. You must install the retail version of the product.
---------------------------
OK
---------------------------

*BOGGLE*

I guess it is time to start scrounging.
Post #: 1
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 22, '06, 11:07:59 AM   
Rodney

 

Posts: 3695
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
Interesting...
I just don't live in the Win32 world that much to have noticed.
Several of these are Win32 through-and-through binaries.
Some others are Win32 wrappers: the parse the args from Win32-sense to Unix sense and
then invoke the Interix version of the utilities.
pwd.exe I think was a Win32-t-a-t one.

Why MS chose to do this I don't know... I can ask.

(in reply to breiter)
Post #: 2
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 22, '06, 12:50:21 PM   
AgentX

 

Posts: 5
Joined: Jan. 29, '06,
Status: offline
quote:

ORIGINAL: Rodney

I just don't live in the Win32 world that much to have noticed.


[OT]
What's your World then? Just curious.
[/OT]

Regards.

(in reply to Rodney)
Post #: 3
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 22, '06, 12:58:01 PM   
breiter

 

Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
I think Rodney is saying that he lives in the C Shell and never uses cmd.exe as a shell to do anything.

(in reply to AgentX)
Post #: 4
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 22, '06, 12:59:02 PM   
Rodney

 

Posts: 3695
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
The Interix world.

(in reply to AgentX)
Post #: 5
[Deleted] - Mar. 22, '06, 1:22:27 PM   
breiter
Post has been moved to the Recycle Bin at Mar. 22, '06, 1:52:26 PM by breiter
Post #: 6
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 24, '06, 5:28:07 AM   
AgentX

 

Posts: 5
Joined: Jan. 29, '06,
Status: offline
quote:

ORIGINAL: Rodney

The Interix world.


I thought you would say Linux, *BSD or Solaris. Silly me!

(in reply to Rodney)
Post #: 7
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 24, '06, 12:22:17 PM   
Rodney

 

Posts: 3695
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
I do "visit" these other worlds regularly

(in reply to AgentX)
Post #: 8
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 27, '06, 3:00:58 PM   
jerker_back

 

Posts: 68
Joined: Jul. 7, '05,
From: Sweden
Status: offline
quote:

cmd> pwd

try
%SystemRoot%\SUA\bin\pwd

or compile your own mixed-mode pwd like this
// pwd.cpp - simple mixed-mode utility
#include <unistd.h>
#include <stdio.h>

extern "C"
int __cdecl main()
{
    char szBuffer[512] = {0};
    getcwd(szBuffer, 512);
    printf(szBuffer);

    return 0;
}
// optional: add unixpath2win

One thing I don't understand is the first initial CLS when a posix or a mixed-mode app is executed from a console window.

(in reply to breiter)
Post #: 9
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 27, '06, 5:16:42 PM   
breiter

 

Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
That is interesting. I'm kind of stunned that it works without using posix.exe to invoke it. It must itself be a mixed-mode?

cmd> c:\WINDOWS\SUA\bin\pwd
/dev/fs/C/Profiles/breiter

However, I think the issue is still valid.

1) This isn't the same as pwd.exe in SFU 3.5. In SFU 3.5, it did do the unixpath2win conversion.

2) I shouldn't have to go and compile my own pwd.exe or get pwd.exe from sourceforge. They should have continued to bundle pwd.exe, sleep.exe, split.exe, strings.exe, unix2dos.exe, etc.

These utilities are nice for filling in missing behavior in interactive and scripting behavior in the CMD shell. They existed before. What happened to backwards comatibility?!

I don't get the intial clear-screen behavior either, but it has always been that way.

Incidentally, why use C++ and then beat on it to make it C?

//pwd.c - mixed-mode utility to print current directory
//in win32 format.
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <interix/interix.h>
#include <limits.h>

int main()
{
        char psxdir[PATH_MAX], windir[PATH_MAX];
        
        if( getcwd(psxdir, sizeof(psxdir)) == NULL )
        {
                perror( "Unable to get current directory" );
                return errno;
        }
        
        if( unixpath2win(psxdir, 0, windir, sizeof(windir)) != 0 )
        {
                perror( "Unable to convert POSIX directory path to Windows format" );
                return errno;
        }
    
        printf("%s\n",windir);
        return 0;
}



% gcc -R pwd.c -o pwd.exe
% cmd
Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.
cmd> .\pwd.exe
C:\Profiles\breiter\Desktop

< Message edited by breiter -- Mar. 28, '06, 12:27:02 AM >

(in reply to jerker_back)
Post #: 10
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 27, '06, 5:27:19 PM   
breiter

 

Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
Interestingly, a number of the utilities in SUA/bin are apparently mixed-mode, but not mixed-filesystem path string aware. For example, you can use strings from the CMD shell, but you have to pass a POSIX path. Since there's no posixpath2win.exe or anything. So it's not very useful for these to be mixed-mode, just kind of weird.

cmd> "c:\WINDOWS\SUA\bin\strings" C:\WINDOWS\system32\ftp.exe | findstr -i calif
ornia
/dev/fs/C/WINDOWS/SUA/bin/strings: C:\WINDOWS\system32\ftp.exe: No such file or
directory

cmd> "c:\WINDOWS\SUA\bin\strings" /dev/fs/C/WINDOWS/system32/ftp.exe | findstr -
i california
@(#) Copyright (c) 1983 The Regents of the University of California.

< Message edited by breiter -- Mar. 27, '06, 5:29:19 PM >

(in reply to breiter)
Post #: 11
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 28, '06, 12:26:33 AM   
breiter

 

Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
It occurs to me that since mixed-mode binaries can be executed from Win23 without a controlling terminal or psxrun.exe, there is a huge opportunity to eliminate the double-up of software packages. I'm sure other people are like me and have win32 versions of a bunch of the same UNIX binaries that are in SUA and/or /Tools.

Some binaries just need to be able to detect and convert win32 paths an be compiled in mixed mode. Then you only need one version instead of two. I'm thinking of things like strings, diff, patch, grep/egrep, zip/unzip, gzip/gunzip, tar, ftp, curl, cadaver, sed, awk, md5/sha1, clamscan/clamd, ghostscript...

Basically any of these could just do double-duty in the Windows world if they were mixed-mode and understood Windows paths. There would be no need for duplicate Windows .exe versions.

Other binaries would work great if they were just mixed-mode. I'm thinking of whois, wait, nice, renice...

This is pretty straightforward to do. For stuff like whois, they just need a recompile with -R.

For stuff that needs to translate Win32 to POSIX paths, there's a bit more work involved, but it isn't a bad port. I tried this out on gnu strings from binutils 2.16 as a proof of concept.

% diff -u strings-ori.c strings.c
--- strings-ori.c       Mon Oct 18 10:04:40 2004
+++ strings.c   Mon Mar 27 23:57:21 2006
@@ -113,6 +113,12 @@
 #define file_stat(f,s) stat(f, s)
 #endif

+/* Interix is the UNIX subsystem for Windows NT, as such it has a unique
+relationship with Win23 paths. */
+#ifdef __INTERIX
+#include <interix/interix.h>
+#include <limits.h>
+#endif
 /* Radix for printing addresses (must be 8, 10 or 16).  */
 static int address_radix;

@@ -380,6 +386,30 @@
 {
   statbuf st;

+/* In the world of Interix we can convert Win32 paths to UNIX-style */
+  #ifdef __INTERIX
+  int winpath = FALSE;
+  char c;
+  char * srch = file;
+  while( (c = *srch++) != '\0' )
+  {
+         if( c == '\\' )
+         {
+                 winpath = TRUE;
+                 break;
+         }
+  }
+  if( winpath )
+  {
+         char buff[PATH_MAX];
+         if ( winpath2unix( file, 0, buff, sizeof(buff) ) != 0 )
+         {
+                 perror( "Unable to convert Windows path to POSIX" );
+         }
+         file = buff;
+  }
+  #endif
+
   if (file_stat (file, &st) < 0)
     {
       if (errno == ENOENT)

After ./conigure --host=i586-pc-interix5.2 and make...

% ./strings --version
GNU strings 2.16
Copyright 2005 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License. This program has absolutely no warranty.
% ./strings /dev/fs/C/WINDOWS/system32/ftp.exe | egrep -i california
@(#) Copyright (c) 1983 The Regents of the University of California.
% cmd
Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

cmd> .\strings C:\WINDOWS\system32\ftp.exe | findstr -i california
@(#) Copyright (c) 1983 The Regents of the University of California.

Cool!

(in reply to breiter)
Post #: 12
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 28, '06, 7:21:13 AM   
jerker_back

 

Posts: 68
Joined: Jul. 7, '05,
From: Sweden
Status: offline
quote:

Incidentally, why use C++ and then beat on it to make it C?

Ha, ha - incidentally, why use C when you can use C++?
If you haven't noticed: I'm a CPP guy - I never write C!
The reason for extern "C" is that the real entrypoint function mainCRTStartUp (in MSCRT) is calling a C-function main. This main must also be called with the __cdecl calling convention, hence __cdecl. We don't have any information on the Interix startup code, but by doing some tests I concluded that the same applies to Interix. If you ever have used the __stdcall calling convention (WIN32 standard), you know why that little addition of __cdecl is there.

So beeing the ATL-freak I am, I would write a bigger app in a ATLish spirit like this:
class CInterixModule
{
public:
    CInterixModule(void) {}
    HRESULT Run(void);
protected:
}

// main app object
CInterixModule _IxModule;

extern "C"
int __cdecl main()
{
    return (int)_IxModule.Run();
}

// Now in C++
HRESULT CInterixModule::Run(void)
{
    ...
}
// class implementation

But that's only coding style
quote:

I'm kind of stunned that it works without using posix.exe to invoke it.

Again, here's some dark undocumented areas. We don't know how posix and mixed-mode apps are executed by the console window. We don't even know what the difference really is between posix and mixed-mode apps. Since we ultimately are in a WinNT environment, there could not be so much "magic" about it. Consider the Korn Shell: It must have a HWND (the NT console window handle) and a HDC for painting (the NT console HDC). It would be good if someone at MS could publish an in depth article about the NT subsystems.

(in reply to breiter)
Post #: 13
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 28, '06, 11:51:59 AM   
Rodney

 

Posts: 3695
Joined: Jul. 9, '02,
From: /Tools lab
Status: offline
> If you ever have used the __stdcall calling convention (WIN32 standard),
> you know why that little addition of __cdecl is there.

It's not so much Win32, ratherwhat the OS expects.
You can also see the __cdecl in the Interix header files.
So there's no secret about it.

> Consider the Korn Shell: It must have a HWND (the NT console window handle)
> and a HDC for painting (the NT console HDC).

Actually, no.
And this is true for all Interix applications.
In Win32-land you have to do this crap.
In the Unix world applications communicate through their tty. The tty "driver"
handles the interfacing to the Win32 GUI for the ttyn* terminals.
If it's an X11 app then the X11 API's are used and it's up to the X11 server
to do the drawing -- the client is also insulated from the non-portable.

(in reply to breiter)
Post #: 14
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 28, '06, 4:18:12 PM   
woehlkmp

 

Posts: 102
Status: offline
> Actually, no

At some point, something has to have an HWND/HDC/etc, although I suspect that "something" is posix.exe itself.

On a different note, I noticed people talking about seamless path conversion, and am having related problems with unixpath2win/winpath2unix, namely that they are far more stupid than Cygwin's cygpath utility (or its cygwin_conv_to_{win32|posix}_path API's) which don't give you garbage if you throw an already-converted path at them.

Thus, I am trying to write my own versions of Cygwin's functions, and it looks like the best way to do this is going to be to implement converters purely from scratch. Unfortunately, to be able to do that, I need to be able to look at the mount points. However, it occurs to me that perhaps I am over-thinking; does Interix even support mounts other than / and the implicit /net/* and /dev/fs/* ones? (In other words, am I OK hardcoding these two cases and relying on symlink resolution?)

If I can, I'll share what I come up with. The advantage would be that you should be able to throw such functions at any path, and they will give it to you in the requested format regardless of what format it was orignally in.

(in reply to Rodney)
Post #: 15
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 28, '06, 8:10:27 PM   
woehlkmp

 

Posts: 102
Status: offline
Actually, it isn't as hard as I thought (although I might be missing corner cases). I came up with the following to convert a path in an unknown format to a requested (POSIX or Windows) format:

static int interix_conv_2psx( const char* path, char* buf, int len )
{
  /* Resolve 'path' into an absolute path */
  if ( ( isalpha(path[0]) && ':' == path[1] ) || ( '\\' == path[0] ) ) {
    /* Path is a Windows path, and can be fed to winpath2unix */
    return winpath2unix( path, 0, buf, len );
  }
  else {
    /*
     * Path is either a POSIX path or is not absolute, therefore we "should" be
     * able to evalute it as a relative unix path to get the correct result.
     * We could do this by round-tripping it, but let's do it the POSIX way. :)
     */
    if ( NULL == realpath( path, buf ) )
      return errno;
    else
      return 0;
  }
}

static int interix_conv_2win( const char* path, char* buf, int len )
{
  /* Resolve 'path' into an absolute path */
  if ( ( isalpha(path[0]) && ':' == path[1] ) || ( '\\' == path[0] ) ) {
    /*
     * Path is a Windows path, and should be fully resolved instead. To do
     * this, we'll use the cheap trick of round-tripping it, because Windows
     * API's are a bitch.
     */
    int err = winpath2unix( path, 0, buf, len );
    if ( 0 == err ) {
      char *tmp = stralloc( len );
      memcpy( tmp, buf, len );
      err = unixpath2win( tmp, 0, buf, len );
      free( tmp );
    }
    return err;
  }
  else {
    /* Path can be fed to unixpath2win */
    return unixpath2win( path, 0, buf, len );
  }
}


Hopefully these will help in your endeavours to create mixed-mode applications!

(in reply to woehlkmp)
Post #: 16
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 6:37:47 AM   
jerker_back

 

Posts: 68
Joined: Jul. 7, '05,
From: Sweden
Status: offline
quote:

__cdecl ... It's not so much Win32, ratherwhat the OS expects.

?? - to clarify on calling conventions see MSDN Old New Thing and links
What I really meant was if you ever used __stdcall you know how annoying it is when the __cdecl is omitted but needed (main, qsort callback functions etc). For no apparent reason I usually compile with __stdcall and are therefore a bit finicky about it.
quote:

At some point, something has to have an HWND/HDC/etc, although I suspect that "something" is posix.exe itself.

Exactly, could it be as simple as posix.exe is really a common NT console app?

Also, the initial clear screen command which happens the first time a posix or a mixed mode app is executed from a console window, indicates that something is loaded once for that console window and stays loaded until the window is closed. It indicates that this loaded something is the same for both the posix and the mixed-mode. So, what is the real difference and maybe more interesting the practical difference between a mixed-mode and a posix app? Take for example /bin/ln 91kB and compare with common\ln.exe 109kB - they differ in size but what is the practical difference? How much of the Interix environment is loaded when a mixed-mode app is executed?

(in reply to woehlkmp)
Post #: 17
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 11:15:50 AM   
breiter

 

Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
quote:

ORIGINAL: jerker_back
Exactly, could it be as simple as posix.exe is really a common NT console app?


I'll go out on a limb here and say that posix.exe pretty much has to be a Win32 console application which uses the tty driver to bridge the Win32 world to the posix subsytsem and generally provides an XTerm-like color VT100 emulator. It has to be a Win32 application because Win32 controls the video output, but since it is also a console application csrss.exe automagically creates a console window for it. I think there was a time when the boundaries between the subsystems were extremely rigid such that posix applications couldn't invoke win32 applications and vice-versa.

FYI. It doesn't seem to matter if something is mixed-mode or not. Any posix binary can be invoked from a CMD console without using posix.exe in R2. It wasn't like that before was it?

(in reply to jerker_back)
Post #: 18
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 11:31:44 AM   
breiter

 

Posts: 343
Joined: Jun. 14, '04,
From: Washington, DC
Status: offline
quote:

ORIGINAL: woehlkmp
However, it occurs to me that perhaps I am over-thinking; does Interix even support mounts other than / and the implicit /net/* and /dev/fs/* ones? (In other words, am I OK hardcoding these two cases and relying on symlink resolution?)


You can't use Interix to mount filesystems. It is done via diskmgmt.msc. There are, unfortunately, two kinds of symbolic links: the NTFS junction points (made with linkd.exe) and POSIX symlinks (made with ln). Oh, and then there are .lnk files which Explorer uses. Win32 doesn't understand the POSIX symlinks, which appear to be 1kb system files in Explorer, but Interix does understand junction points--or rather I suspect that they exist at a layer of abstraction beneath Interix so that the junction points appear to be regular directories to Interix.

On my computer, for example, C:\Programs is a junction point that points to C:\Program Files.

cmd> dir C:\
...
03/22/2006 10:30 PM <DIR> Program Files
09/26/2005 07:00 AM <JUNCTION> Programs
...

% ls -l /dev/fs/C
...
drwxrwx---+ 1 +Administrators +SYSTEM 0 Mar 22 22:30 Program Files
drwxrwx---+ 1 +Administrators +SYSTEM 0 Mar 22 22:30 Programs
...

In any case, however it works, winpath2unix(2) resolves junction points to proper posix paths.

(in reply to woehlkmp)
Post #: 19
RE: Several "common" utils gone in 5.2; 3.5 v... - Mar. 29, '06, 12:00:45 PM   
woehlkmp

 

Posts: 102
Status: offline
What I ended up doing is: if the path starts with '\' or '<letter>:', then I treat it as a Windows path; otherwise I treat it as a POSIX path (either a relative path, which should be relative to the Interix working directory, or a POSIX-style absolute path). I think this bypasses the 'junction point' stuff because winpath2unix knows how to handle them.

The problem I was trying to get around is that winpath2unix, by itself, treats '/usr' like '\usr', which results in things like '/dev/fs/C/usr' rather than '/usr' like I wanted! (Similarly, 'unixpath2win C:\\' just doesn't work; again, not what I want.)

This brings up an interesting observation... if I do 'touch foo ; ln foo bar' on an NTFS drive, then Windows outside of POSIX recognizes the hard link... but 'dir' doesn't list it as '<JUNCTION>'. Is a junction point specific to directories; thus a 'third relative' to hardlinks (which NTFS seems to support?) and symlinks (which are not understood outside of Interix)? Would it perhaps be fair to say that a "junction point" is roughly equivalent to a 'mount --bind'?

(in reply to breiter)
Post #: 20
Page:   [1] 2 3 4   next >   >>
All Forums >> [SFU / Interix / SUA Technology] >> Windows Server 2003 R2 SUA >> Several "common" utils gone in 5.2; 3.5 versions unusable Page: [1] 2 3 4   next >   >>
Jump to:





New Messages No New Messages
Hot Topic w/ New Messages Hot Topic w/o New Messages
Locked w/ New Messages Locked w/o New Messages
 Post New Thread
 Reply to Message
 Post New Poll
 Submit Vote
 Delete My Own Post
 Delete My Own Thread
 Rate Posts


Search All Forums -

Advanced search


SPONSORS



Forum Software © ASPPlayground.NET Advanced Edition 2.5 ANSI

0.078