Live PowerShell With Karl Prosser

The other day I was working with a ne wPSV3 feature called Scheduled Jobs which is really cool as it allows you run some stuff as a Scheduled job, which uses the Scheduled Task engine to run it. You can set triggers like on start up, but unfortunately, and this is something I am frustrated about, not “immediately”. However you can trigger it to say run in 5 seconds which is long enough to ensure even a slow computer is going to get to it before the time expires and it would never .

So often I’ll do a demo like

Register-ScheduledJob -name quicktest -ScriptBlock { 1; sleep 5; 2} -Trigger (New-JobTrigger -Once -At (get-date).AddSeconds(4))
sleep 6
get-job quicktest | wait-job | Receive-Job

 

What this does is register a scheduled Job, which triggers in a few seconds, sleep to make sure it gets triggered, then uses get-job , which gets the job INSTANCE of the scheduled job, and does the normal stuff to wait and receive the data.

This works all good and dandy. Other than when it doesn’t, which according to the laws of DEMO, happens in a demo. So I ran something like this, and the job never seemed to be triggered. I tried again and again and I just couldn’t get it to work. and from the PowerShell side of things there just was no indication of error. it is as if I had never tried to run it.

 

So I started poking around in Task Scheduler. PowerShell stores the Scheduled tasks at \Microsoft\Windows\PowerShell\ScheduledJobs

 

Aha found the culprit. I was taking my laptop somewhere else to do the demo and had plenty of battery, However by default (not a bad thing) scheduled Tasks won’t launch when on battery.

It is a shaming pity that PSV3 doesn’t capture the error and create and failed job instance with the error. that would be definitely useful, and consistent with the PowerShell Job experience in general. At least we should have a Cmdlet that can show us warnings and errors for our scheduled jobs.

There are many reasons why a scheduled task can’t launch , from permissions, to this battery thing, to concurrency issues (i.e by default the policy only allows once instance of a scheduled task to run at once)

In my sample I could have just updated the policy to allow it to run on battery (-ScheduledJobOption (New-ScheduledJobOption -StartIfOnBattery), but still the point is it failed, and there was no indication of why. So be aware of these situations and work around them.

Register-ScheduledJob -name quicktest -ScriptBlock { 1; sleep 5; 2} `
-Trigger (New-JobTrigger -Once -At (get-date).AddSeconds(4)) `
-ScheduledJobOption (New-ScheduledJobOption -StartIfOnBattery)
sleep 6
get-job quicktest | wait-job | Receive-Job

There is an issue that can basically prevent you from calling certain dotnet apis.

Basically the trick is just to make a class that’s .tostring() returns a null, and then when you pass that to an api through powershell that takes a string powershell will convert the class to a string() thus calling your code that returns a null..

namespace testnullD
{
public class nulltest
{
public nulltest(){}
public override string ToString() {return null;}
}
}

Here is a complete example you can use.

$a = @"

using System;
namespace ClassLibrary1
{
#proof of concept
public static class Class1
{
public static string Foo(string value)
{
if (value == null)
    return "Special processing of null.";
else
    return "'" + value + "' has been processed.";
}
}
}
"@

$b = @"
using System;
namespace testnullD
{
public class nulltest
{
public nulltest(){}
public override string ToString() {return null;}
}
}
"@
add-type $a
add-type $b
$psnull = new-object testnullD.nulltest
add-type $a
[ClassLibrary1.Class1]::Foo($null)
[ClassLibrary1.Class1]::Foo($psnull)

Given the context on the last few posts. I’ve made a simple helper method in C# that can take a simple powershell hashtable and create a PSCustomObject based on it. Here is an example of how you can call it.

[snapinini.newobjecthelper]::newObjectFast(@{name="karl";age=31;now = [datetime]::Now})

very simple and its at least 6 times faster than the closest other technique in powershell. Most of hte overhead is in creating the hashtable. (otherwise its 90 times faster). The hashtable syntax is very convenient however maybe even its overhead is too much and we deserve a better way.

Here is the C# method. Its pretty basic stuff.

public static PSObject newObjectFast( Hashtable noteproperties )

        {

            PSObject obj = new PSObject();

            if (noteproperties != null)

                foreach(DictionaryEntry item in noteproperties)

                    obj.Properties.Add(new PSNoteProperty((string)item.Key,item.Value));

            return obj;

        }

- Karl

Based on the performance testing and work being done by myself , mow , Brandon Shell and others the question has come up what is the quickest way to generate a PSCustomObject, whether in script , on in C#, and how do you even do it in C#?

Some typical ways of doing in PowerShell have been either something like the following trick:

$a = 1 | select a , b , c , d

or with using the Add-member cmdlet as below:

$a | add-member -membertype noteproperty -name status -value done

however both are really slow. There is another way to do it in script, and that is directly with the PSobject, but still PowerShell (especially version 1) has a huge overhead when creating new objects with new-object but below is an example none the less.

$obj = new-Object system.Management.Automation.PSObject

$note = new-object System.Management.Automation.PSnoteproperty "karl" , "a value"

$obj.psobject.members.add( $note );

But if you really want to do speed, or if you have need to generate these objects in a cmdlet regardless of speed here is how you can do it in C#.

public static PSObject newPsCustomObject2()

        {

            //Creating a PSobject without any parameters in the constructor creates a PSCustomObjectg

            PSObject obj = new PSObject();

            //PSNoteProperties are not strongly typed but do contain an explicit type.

            obj.Properties.Add(new PSNoteProperty("age",24));

            obj.Properties.Add(new PSNoteProperty("name","jon"));

            //Alias allow you to cast one property as another as well as just plain aliasing

            obj.Properties.Add(new PSAliasProperty("ageasstring","age",typeof(string)));

            return obj;

        }

Some notes about the above, I didn’t show how you could add ScriptProperties or ScriptMethods yet as I wanted to keep it simple and not cover creating scriptblocks. Before I go lets performance test this baby compared to creating this in pure PowerShell.

So in the:

1) first corner we have creating the object with select-object, then setting the properties.(but we can’t set the alias)

2) Creating the object with new-object, then using add-member to add all the properties.

3) calling our static method above.

#select-object version

for($i = 0;$i -lt 50000;$i++) { $a = 1 |Select-Object age, name; $a.age = 24; $a.name = "john" }

#add-member version

for($i = 0;$i -lt 50000;$i++)

  { $a = new-Object psobject;

    $a | add-member -membertype noteproperty -name age -value 24

    $a | add-member -membertype noteproperty -name name -value "john"

    $a | add-member -membertype aliasproperty -name "ageasstring" -value "age" -secondvalue "string"

  }

#Our fast version

for($i = 0;$i -lt 50000;$i++) { $a = [snapinini.newobjecthelper]::newPsCustomObject2(); }

so we are creating 50,000 objects and the results are.

select-object	27.15 seconds.
add-member	118.79 seconds
C# method	1.9 seconds.

and i used to complain with the performance hit of a address lookup for a method in a C++ virtual method table!!

at worst the official add-member technique is over 60 times slower than calling C#, and thats not really RAW C#, thats still powershell invoking a C# method 50,000 times, and powershell managing a loop.

if fact just to test i added another test

$a = [snapinini.newobjecthelper]::newPsCustomObject3();

where newPSCustomObject3() moves the 50,000 loop to inside C#.. and here the speed is 0.95 seconds, so about half the speed of invoking the C# method from powershell each time – this speed difference I am happy with though. The PowerShell team seem to have done a good job of invoking C# methods quickly.

Another day I should investigate the speed of creating objects that have been created with the extended type system.

I’d love if somebody can run these tests in v2.

Also sometime soon I shall share a helper function you can use to quickly generate a PScustomObject with properties and values you specify in script.

-Karl

I’ve spend a lot of time in the past looking at the performance of PowerShell but have never gotten around to blogging about it. However it has become quite a hot button among MVPs and others in recent days and we’ve been busy testing and comparing the speeds different techniques in PowerShell in both V1 and V2.

In V1 there seems to be extreme slowdowns when invoking cmdlet, functions, any type of scriptblock and when using the pipeline. If you write a one liner that is easy to read, very elegant, but uses a lot of “powershelly” techniques its performance is often terrible, compared to it you write it out as a longer script using more traditional imperative style with “C”ish loops as the like. This brings up the whole concept of what is cheaper and more valuable – CPU cycles that keep getting cheaper, or human time in writing and maintaining code – dynamic languages and especially shells will definitely answer that programmer productivity is more valuable, and thus performance can come second.(read about it in the context of python here.) However 59 millisecond average to call an empty powershell function on a dual core machine is pushing this truth to an absurdity. PowerShell should as a language should NOT be slower than BASIC on my Commodore-64 – an 1MHZ 8 bit machine.

Here is some background of where we are coming from. Brandon Shell Started it all here. And after we spend hours running tests and bantering about the subject on the #powershell IRC room on freenode, Mow covered some cool stuff here.

Basically the results of my tests showed that new-object is very slow in V1, but calling any CMDLET, even a simple C# cmdlet with one line of code is extremely slow, and about 30 to 50 times slower than calling that SAME  line of code from powershell as a static method on a C# assembly. Additionally if you take that 50 times faster static method call, and wrap in it a simple powershell function, you loose all your spend gains and its even slower than calling the new-object CmdLet.

The great thing from Mows tests is that in V2 it seems that on the language side of things, scriptblocks,functions and the pipelines have spend up drastically (in the magnitude of about 2000% to 3000%. And cmdlet invocation has spend up also, but is about 6 times slower than just calling C# static methods. GO V2 GO V2 , but is it enough?

Given that V1 is going to be important for a long time, I think that powershell scripters need to still try to script in a “PowerShelly” way, but when they determine that something really has to be performant , they need to know what workarounds they can do in PowerShell to speed things up, understanding where the bottlenecks in PowerShell are.

But this post isn’t about those performance solutions – I’m sure that we will cover a lot of them over the coming of the months. I want to RANT.

One of the reasons I think PowerShell has so many language quirks (and thus related gotchas when learning to do advanced stuff and solving bugs) is because it is Defined by Implementation instead of being Defined by Specification. To be honest, PowerShell is more of a platform , than a language or even a shell, and as it brings some new ideas and melds existing ideas in totally new ways, almost experimental ways, I don’t think it could have been defined any other way. However its quirks and yes also its performance is a result of that. I believe it needs to be reviewed by traditional language designers and not be excused “because its a shell language”. I believe that the importance of PowerShell is growing phenomenally and being part of Microsoft’s common engineering requirement I strongly believe such close working with the CLR team should be done, and even if PowerShell is defined by Implementation and will likely be in version 2 also – SPEND THE TIME TO AT LEAST DEFINE THE SPECIFICATION, DOCUMENT THE INTENDED (and thus real) BEHAVIOUR, RELEASE A FULL GRAMMAR, RELEASE OFFICIAL XSD SCHEMAS of the XML ELEMENTS, and if specification is what defines it, and that specification is best represented in the automated unit tests – maybe , just maybe even release such tests  , as per other dynamic languages. I think some of this stuff should at least be done BEFORE V2 is released. I mean getting a V2 and not even having a V1 grammar. </end of rant>

Disclaimer:

Despite the above concerns, I love PowerShell, and the general performance doesn’t bother me. Being able to code at the speed of thought, and do things with far less code than C#, and do things interactively and dynamically totally win out. PowerShell does introduce many new tangents on concepts and mash together some amazing things, its amazing they got it as right as they did the first time. I know if I had implemented such a platform/language it would be less consistent and probably slower than a ZX81. PowerShell V2, even though only in CTP2 already shows great performance enhancements.

If that disclaimer gave you the warm fuzzies, don’t get too comfortable, i have another pet peeve to blog about, and that is the fallacy called “Restricted Runspaces” and how the lack of being able to truly sandbox PowerShell has me running in the direction of IronRuby – But that’s for another day.

-Karl / Shell Tools