Live PowerShell With Karl Prosser

Todays Blog entry is going to cover the HOW of how I put together my demos of Portable PowerShell. Tomorrow I’ll move back to clearly defining what I’d like to see from Microsoft. But before I get to the full details a little background:

Background

So what’s stopping PowerShell or practically anything from being portable? dependence on a particular file structure and the registry. Basically the technique I  "HOOK" apis so that whenever the process tries to call certain APIs (the filesystem and registry ones) the code in memory is changed to jump to custom code instead of the address of the DLL function export where DLL has been loaded. From there custom checks are done to see if its a file or registry setting to do with PowerShell, and if so the code delivers it up, as if it were there, even though its not.

This is a technique that as a programmer I have been doing for a long time. It involves a lot of assembler, historically being able to inject a DLL into a process and in memory rewrite the code, but it works great. The first time I worked out this technique as a programmer was back in the days of windows 95 which wasn’t unicode. I made an application that could enable Chinese and Japanese characters on english windows. Effectively I hooked the TextOut API, and if it was a process that had been chosen to say display in Japanese, and the font chosen was one that was chosen for this purpose then I custom drew the Japanese or Chinese Glyph otherwise I just called the real TextOut API call. Another technique when working on VisualJockey many years ago, I wanted to be able to pipe the output on a standalone graphics program into VisualJockey so we could do some video processing on it. The problem is that app drew to fullscreen directX itself, so I had to "hook" the directX apis, so that when it created the surface it didn’t make it fullscreen, and when it rendered it rendered to Texture instead of screen, and then some extra code to retrieve what it had rendered. Another Situation was changing a DB backend of an application without the Applications source, *simply* by hooking the ODBC apis (where we also could discover the password used). Another thing I used this technique for was to add some functionality to abandoned freeware that had no source code.

Fast forward to about 2002, Josua Honger , the main architect of VisualJockey, and also the Hippotiser (I could talk a week just about these amazing programs, the Hippotiser has gone on to become a huge sensation powering realtime effects for live T.V, large concerts, events like the nobel peace prize, broadway shows, and the worlds largest audiovisual setup- Love by Cirque de Soliel in Las Vegas)  were wanting to use the engine of VisualJockey to make a demoscene demo , without the copy protection , and in a portable manner (Demoscene demos aren’t allowed to install anything), and on top of that not scare the Green Hippo (who recently purchased the VisualJockey engine and didn’t want to risk it getting out and reused). We didn’t have time to roll our own solution so we found a rather recent product called Thinstall. Thinstall used the techniques mentioned about, but it stored all the files you choose in its own encrypted virtual file system inside one file - Just what we were needing. Thinstall in those days also was cheap enough for us, even though it was a hobby project, but as a young product it didn’t quite work fully with our VisualJockey because we used some interesting techniques anyhow, but I did gain a lot of respect for it, and for its author Jonathan Clark. Over the years I’ve played with it a few times for some hobby projects , but the cost went up and up and I couldn’t justify using it.

Fast forward to 2008, the complexities and risks of deploying and configuring applications have become a real pain point for many corporations and an abundance of solutions have tried solving this. Thinstall evolved its PR to target this market - and named it Application Virtualization, and most recently got acquired by VMWARE. (and renamed ThinApp) So now that there was a great beta to try. the scary thing is HOW EASY THIS WAS. I expected that I would have to spend many hours manually tweaking what to include/exclude - setting up the behaviour of the registry virtualization, but I had something done in 20 minutes, pretty much just following a few instructions. and Click next / next /next

HOW HOW HOW HOW HOW HOW HOW HOW

So I wanted to make 4 different versions of Portable Powershell

• Powershell V1 (just powershell, don’t include dotnet 2.0 - thus dotnet 2.0 is needed to be installed on a machine for it to work)
• Powershell V2 CTP2 (just powershell, don’t include dotnet 2.0 - thus dotnet 2.0 is needed to be installed on a machine for it to work)
• PowerShell V1 (include the who dotnet 2.0 so it can run even where dotnet isn’t)
• PowerShell V1 (include the who dotnet 2.0 so it can run even where dotnet isn’t)

The Key this is to have a Fresh OS to start from, so that you don’t miss any important files , and don’t get rubbish either, so the place to build this is most definately in a VM.

So what did i do?

1. Predownloaded what was needed
1. XP SP2 install CD
2. VMWARE THINAPP BETA (plus free beta key)
3. DOTNET 2.0 redistributable
4. Powershell V1 for XP installer
5. Powershell V2 CTP2 installer
2. Setup the OS in a VM.
3. Installed ThinAPP
4. Made a snapshot of the VM
5. started the thinapp process
6. Installed dotnet 2.0
7. Installed PowerShell V2CTP2
8. Copied over PowerShell Analyzer
9. Completed the thinapp process , and build the Portable EXE for my V2 with dotnet Portable Powershell
10. Copied my portable powerShell off the VM
11. Reverted the snapshot
12. Repeated the process but for PowerShell V1
13. Reverted the snapshot
14. Installed dotnet 2.0
15. Made a new snapshot
16. repeated the process but just installing Powershell V1 (to make a smaller less heavy portable powershell that didn’t contain dotnet 2.0)
17. reverted and repeated the process for Powershell V2

There there you have it, 4 different flavors on a Portable Powershell made in just a few hours. I can’t give them enough praise for how far Thinstall has come and how easy the process was.

Are there any other options?

Definately! Thinstall is not the sole player in this now "application virtualization" arena. The management , configuration and deployment has been bundled into apps and called "Application Streaming". Citrix added Application Streaming, Symantec has Altiris, and Microsoft bought out SoftGrid. The "streaming techniques aren’t typically really portable since they have some sort of agent or client/server architecture but the interesting one in this mix is software, Which i think is now renamed Microsoft Application Virtualization or APP-V for short. You can read their blog here. Actually the release candidate was just released YESTERDAY.

With thinstall, I’m interesting what the pricing is going to be under VMWARE. When its in beta its definitely financially feasible for me to have my own personal portable powershell for personal use, but that’s probably not the case post RTM.

I didn’t have time to play with APP-V yet. SO MUCH CHALLENGE IS WILL SOMEBODY TRY TO VIRTUALIZE POWERSHELL WITH APP-V AND TELL US HOW IT GOES. I presume Microsoft will be more likely to allow redistribution of PowerShell through one of their own technologies

Another cheap option I would challenge somebody is to try it in sandboxie , see if sandboxie is up to the job, as its very cheap, but i don’t know if it does full "portable"

Finally there is another way, that I might implement, and that is roll my own. I wouldn’t include the dotnet framework in this one (just too much work), but I can whip up a DLL in C++ and Assembler in no time to take care of powershell’s filesystem and registry needs. Of course I couldn’t redistribute portable PowerShell but I could make it into a small PORTABLEPOWERSHELLBUILDER.EXE app, that has a button that will build a portable PowerShell for you on any computer that has PowerShell installed in a similar manner to how BartPE or ReatogoXPE can build a bootable XP CD based on the files on a particular OS install.

Apart from the challenge of doing that, and doing some fun techniques that I’ve almost forgotten, and as a PowerShell tool vendor and partner of MS, I don’t want to do that. I want MS to allow PowerShell to be redistributed, in a portable manner, and if they can fit it into their schedule, do some minor reachitecture of PowerShell V2 to be naturally portable, but more on that in my next blog post.

Over and Out,

Karl Prosser / ShellTools

Given the context on the last few posts. I’ve made a simple helper method in C# that can take a simple powershell hashtable and create a PSCustomObject based on it. Here is an example of how you can call it.

[snapinini.newobjecthelper]::newObjectFast(@{name="karl";age=31;now = [datetime]::Now})

very simple and its at least 6 times faster than the closest other technique in powershell. Most of hte overhead is in creating the hashtable. (otherwise its 90 times faster). The hashtable syntax is very convenient however maybe even its overhead is too much and we deserve a better way.

Here is the C# method. Its pretty basic stuff.

public static PSObject newObjectFast( Hashtable noteproperties )

        {

            PSObject obj = new PSObject();

            if (noteproperties != null)            

                foreach(DictionaryEntry item in noteproperties)

                    obj.Properties.Add(new PSNoteProperty((string)item.Key,item.Value));            

            return obj;

        }

- Karl

Based on the performance testing and work being done by myself , mow , Brandon Shell and others the question has come up what is the quickest way to generate a PSCustomObject, whether in script , on in C#, and how do you even do it in C#?

Some typical ways of doing in PowerShell have been either something like the following trick:

$a = 1 | select a , b , c , d

or with using the Add-member cmdlet as below:

$a | add-member -membertype noteproperty -name status -value done

however both are really slow. There is another way to do it in script, and that is directly with the PSobject, but still PowerShell (especially version 1) has a huge overhead when creating new objects with new-object but below is an example none the less.

$obj = new-Object system.Management.Automation.PSObject

$note = new-object System.Management.Automation.PSnoteproperty "karl" , "a value"

$obj.psobject.members.add( $note );

But if you really want to do speed, or if you have need to generate these objects in a cmdlet regardless of speed here is how you can do it in C#.

public static PSObject newPsCustomObject2()

        {

            //Creating a PSobject without any parameters in the constructor creates a PSCustomObjectg

            PSObject obj = new PSObject();

            //PSNoteProperties are not strongly typed but do contain an explicit type.

            obj.Properties.Add(new PSNoteProperty("age",24));

            obj.Properties.Add(new PSNoteProperty("name","jon"));

            //Alias allow you to cast one property as another as well as just plain aliasing

            obj.Properties.Add(new PSAliasProperty("ageasstring","age",typeof(string)));            

            return obj;

        }

Some notes about the above, I didn’t show how you could add ScriptProperties or ScriptMethods yet as I wanted to keep it simple and not cover creating scriptblocks. Before I go lets performance test this baby compared to creating this in pure PowerShell.

So in the:

1) first corner we have creating the object with select-object, then setting the properties.(but we can’t set the alias)

2) Creating the object with new-object, then using add-member to add all the properties.

3) calling our static method above.

#select-object version

for($i = 0;$i -lt 50000;$i++) { $a = 1 |Select-Object age, name; $a.age = 24; $a.name = "john" }

#add-member version

for($i = 0;$i -lt 50000;$i++) 

  { $a = new-Object psobject;

    $a | add-member -membertype noteproperty -name age -value 24

    $a | add-member -membertype noteproperty -name name -value "john"

    $a | add-member -membertype aliasproperty -name "ageasstring" -value "age" -secondvalue "string"    

  }

#Our fast version

for($i = 0;$i -lt 50000;$i++) { $a = [snapinini.newobjecthelper]::newPsCustomObject2(); }

so we are creating 50,000 objects and the results are.

and i used to complain with the performance hit of a address lookup for a method in a C++ virtual method table!!

at worst the official add-member technique is over 60 times slower than calling C#, and thats not really RAW C#, thats still powershell invoking a C# method 50,000 times, and powershell managing a loop.

if fact just to test i added another test

$a = [snapinini.newobjecthelper]::newPsCustomObject3();

where newPSCustomObject3() moves the 50,000 loop to inside C#.. and here the speed is 0.95 seconds, so about half the speed of invoking the C# method from powershell each time - this speed difference I am happy with though. The PowerShell team seem to have done a good job of invoking C# methods quickly.

Another day I should investigate the speed of creating objects that have been created with the extended type system.

I’d love if somebody can run these tests in v2.

Also sometime soon I shall share a helper function you can use to quickly generate a PScustomObject with properties and values you specify in script.

-Karl

One thing that has bothered me about true cmdlets in contrast scripts is that they lived in SnapIns and couldn’t be deployed with xcopy as scripts can because SnapIns required Registration/Installation. I build my own way to deal with this need some time ago, and thought that I’d start a blog series covering it. I call them Snapininis or SnapIn-Lites. A good thing is you can achieve the same end result in PowerShell V2 with modules - but that is V2 and a while away from being released, and then even further away from being fully deployed.

So how does this work?

1) you have to load an assembly into memory in powershell that contains your snapin.
2) you add the cmdlets directly into the runspace that is running and tell powershell to update its list of cmdlets.

Here is the C# code for a sample barebones SnapInini.

using System;

using System.Collections.Generic;

using System.Text;

using System.Management.Automation;

using System.Management.Automation.Runspaces;

namespace Snapinini

{

    public class DynamicCmdLet

    {

        public static void Load()

        {

            CmdletConfigurationEntry conf = new CmdletConfigurationEntry("dummy-cmdlet", typeof(ForeachDictionaryCmdLet), null);

            Runspace.DefaultRunspace.RunspaceConfiguration.Cmdlets.Append(conf);

            Runspace.DefaultRunspace.RunspaceConfiguration.Cmdlets.Update();

            Runspace.DefaultRunspace.CreateNestedPipeline(

                "new-alias dcmd dummy-cmdlet; write-host ’snapinini loaded’", 

                false).Invoke();

        }

    }

    [Cmdlet("dummy", "cmdlet")]

    public class ForeachDictionaryCmdLet : System.Management.Automation.PSCmdlet

    {

        protected override void EndProcessing()

        {

            WriteObject("hello from the dummy cmdlet");

        } 

    }

}

In the above sample I have a very very simple cmdlet. You can see that i get the defaultrunspace, go into its RunspaceConfiguration and add the cmdlets and update the list. I am additionally running some powershell script that adds an alias to the cmdlet and writes a message to the screen just as an example showing you that you can do more stuff if you wish. In the above code the load is a static method so its easy to call from powershell once you’ve loaded the assembly, but in all honesty the Load() code could just as well have been written in the calling PowerShell.

So how do we load and use this in PowerShell?

[system.Reflection.Assembly]::LoadFrom("snapinini.dll")

[snapinini.DynamicCmdLet]::load()

dummy-cmdlet

dcmd

Where to from here? This example is very simple, you could easily add multiple cmdlets into the equation, specify help files, add custom type definitions etc. Another thing I had experimented with is taking a real snapin and using reflection to see the cmdlets in it and adding them directly, however unless you know exactly what a snapin does this is dangerous because in its registration it might set up a bunch of other things also that are important.

Another thing I should cover next time is a safe way to load assemblies from any location and ensure that all their references get resolved seamlessly.

-Karl