Aaronontheweb

Hacking .NET and Startups

Intro to Node.JS for .NET Developers

December 14, 2011 04:51 by Aaronontheweb in Azure, Node // Tags: , , , // Comments (7)

Node.js darkMicrosoft announced out-of-the-box support for Node.JS on Windows Azure on Tuesday; we pushed both an official Node.JS SDK for Windows Azure services (table storage, blob storage, and queues) and some tools for creating and managing Node.JS roles on Windows Azure, both of which are available through the official Windows Azure page on Github.

I’ve been playing around with Node for a short while and have the pleasure of working with some great startups that utilize Node heavily in production, so I wanted an opportunity to explain to my fellow .NET developers why they should be excited about Node support on Azure and how they can put it to work for them.

Node at a Glance

If you want to spend 90 minutes and get a full 100-level understanding of Node, check out The Node Beginner Book. If you’re fine with the footnotes, keep reading.

Node is an asynchronous distributed programming platform built on top of Chrome’s V8 JavaScript engine (the same engine used to parse and execute client-side JavaScript inside Chrome.) Node is actually server-side JavaScript, but its syntax and prose are familiar to every web developer to some extent.

Node’s true innovation is its evented + asynchronous I/O model.

var http = require('http');
http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end(myJavascriptObject.getSomeStatusInfo());
}).listen(1337, "127.0.0.1");

The primary method of any Node application runs a single-threaded continuous event loop (the .listen method of the HTTP server), which farms out long-running tasks to worker threads (the anonymous function) which callback the main event loop once they’re finished:

nodejs for dotnet

There are three major advantages to this model:

  • The main event loop uses a single thread to handle multiple concurrent connections, which makes the overhead of Node.JS grow relatively slowly as the number of requests it has to serve increases as there’s no OS thread / process initialization overhead;
  • All long-running tasks (network I/O, data access, etc…) are always executed asynchronously on top of worker threads which return the results via callback to the event loop thread; and
  • JavaScript’s language features (functions as objects, closures, etc…) and Node’s programming model make this type of asynchronous / concurrent programming much easier to utilize – there’s no thread management, no synchronization mechanisms, and no message-passing nonsense. This eliminates a lot of pitfalls that most developers fall into when attempting to develop concurrent applications.

Using Node.JS with .NET

So, you’re a veteran ASP.NET MVC or WCF developer… Why should you be interested in taking a look at Node?

First, Node is fully supported in environments .NET developers are familiar with. Node runs on Windows; Node can be run as a managed module inside of IIS; and Node can now run on Windows Azure. There’s even a NPM package for having Node.JS interop with SQL Server.

Second, Node does a great job tackling vertical issues like real-time notifications, web sockets, and other scenarios that aren’t easily addressed through existing .NET technologies (although SignalR is pretty cool.)

Node isn’t a replacement for everything – frankly, it’s not a great solution for heavy web + database CRUD applications or for serving static content.

Here are some scenarios that I like for Node:

  • Handling multiple concurrent connections (web sockets, handling events from real-time messaging systems [ZeroMQ / Azure Service Bus], anything with socket programming…)
  • High-volume, small response-size systems – think of a real-time logging system where you have to write millions of messages an hour and only send small ACK response objects back to the caller – Node.JS is perfectly suited to handle this;
  • Real-time Pub/Sub systems – if you have an application that is heavy on notifications and needs to do it in real-time, Node is a great choice given its penchant for getting in and out of requests quickly.

Need a pragmatic example?

Take a close look at Socket.IO, arguably the best solution for working with web sockets and long-polling available on any platform.

If you have an application where real-time notifications are important like a messaging application or a social game, then running Socket.IO alongside IIS on your boxes gives you a significantly simplified solution both on the client (the Socket.IO client is backwards compatible all the way to IE5.5) and the server for handling socket connections and routing, and it takes a minimal amount of resource overhead to do it.

I’m going to be doing some more work around Node.JS and Azure in the New Year, so if you have any questions or comments please feel free to hit me up in the comments here.

Want to get started with Node.JS and Windows Azure?

If you want to try out Node.JS on Azure, sign up for the Windows Azure free trial and install the Node.JS Azure SDK via Web Platform Installer.

If you enjoyed this post, make sure you subscribe to my RSS feed!



How to Modify Machine.config on Windows Azure Web Roles

December 1, 2011 19:04 by Aaronontheweb in Azure // Tags: , , // Comments (0)

Earlier this week I spent some time helping a company troubleshoot some performance issues with Windows Azure – their ASP.NET request queue was growing longer than the maximum amount of queued requests supported by default in IIS (5000) during bursts of high activity, so I had to help them find a way to empty the queue faster.

One way to do this is to modify the number of max/min IO and worker threads available to any web role instance, and these values can only be modified through machine.config settings.

As you know, Windows Azure VMs are not persistent so you can’t modify machine.config and save it to file system like how you might if you were hosting IIS on your own local box.

Thus, I crafted a solution using AppCmd and Windows Azure startup tasks.

Here’s some of the code below:

REM Increases the number of available IIS threads for high performance applications REM Uses the recommended values from http://msdn.microsoft.com/en-us/library/ms998549.aspx#scalenetchapt06_topic8 REM Values may be subject to change depending on your needs REM Assumes you're running on two cores (medium instance on Azure) %windir%\system32\inetsrv\appcmd set config /commit:MACHINE -section:processModel -maxWorkerThreads:100 >> log.txt 2>> err.txt %windir%\system32\inetsrv\appcmd set config /commit:MACHINE -section:processModel -minWorkerThreads:50 >> log.txt 2>> err.txt %windir%\system32\inetsrv\appcmd set config /commit:MACHINE -section:processModel -minIoThreads:50 >> log.txt 2>> err.txt %windir%\system32\inetsrv\appcmd set config /commit:MACHINE -section:processModel -maxIoThreads:100 >> log.txt 2>> err.txt

I’ve open-sourced the rest of my IIS machine.config / app pool scripts on Github. Take a look at them and let me know if you have any questions.

If you enjoyed this post, make sure you subscribe to my RSS feed!



How to Migrate Data between On-Premise SQL Server 2008 R2 and SQL Azure Without Getting a Migraine

November 28, 2011 17:33 by Aaronontheweb in Azure, SQL Server // Tags: , , // Comments (2)

I love my job at Microsoft, but there are some times when we simply make it really damn hard for people to do business with us. Migrating data from an on-premise SQL Server to SQL Azure is sadly one of those lapses where, for whatever reason, we’ve left people who aren’t full time SQL Server DBAs totally lost in the wilderness with a set of poorly documented and often dysfunctional tools.

After spending a couple of hours shaving yaks myself tonight trying to move a ~250mb data set (small, but not trivially small) from one SQL Azure database to another, I thought it would be best if I documented what I did to make it work.

Scenario: I needed to download an expensive dataset on one SQL Azure account and migrate it to another on a different SQL Azure subscription, but I needed to make and test some schema changes against the data set locally before I exported it back to its final resting place.

Seems simple enough, right? I just need to:

  1. Download the SQL Azure Database to my local system;
  2. Make a back-up of the dataset locally in case I screw up the other one during my schema changes;
  3. Finish making / testing / integrating schema changes;
  4. Deploy to SQL Azure on new subscription.

Sounds pretty easy to me, as it would any sane developer who’s had some experience working with the Microsoft server stack since 2000.

Rather than tell you the things I spent hours trying that don’t work I’ll explain what does.

Before you do any of this, make sure your SQL Azure Firewall has an exception for whichever development / backup machine you’re using.

Backing Up SQL Azure to On-Premise SQL Server 2008 R2: RedGate’s SQL Azure Backup

SQL Azure Backup from RedGate solved my first problem painlessly – currently SQL Azure Backup is free so grab a copy while you still can.

All you have to do is point it to your target database on SQL Azure and your target on-premise SQL Server database (screenshot taken from RedGate.)

That’s about as complicated as it gets.

Pushing from SQL Server 2008 R2 to SQL Azure: SQL Server Management Studio Data Export (Requires Magic)

One of the SQL Azure migration techniques Microsoft recommends is using the SQL Server Import and Export Wizard to push your data to SQL Azure. Unfortunately they leave one critical part out, which I will show you.

You begin by selecting your database in Management Studio (I’m using the Express edition.)

image

And for the second step you can leave all of the client / connection settings as-is for your on-premise SQL Server (shown below.)

image

And now we come to the part that the MSDN documentation totally left out… If you’re like me, you’ll naturally try to login using the SQL Server Native Client 10.0 datasource for your export target, because hey, makes sense right?

image

As it turns out, you need to use the .NET Framework Data Provider for SqlServer and change the following fields:

  1. Set Security –- Encrypt to true;
  2. Set Security – Password to your SQL Azure login password;
  3. Set Security – User ID to your SQL Azure user id without the @servername at the end;
  4. Set the Source – Data Source property to the servername.database.windows.net – no need to specify any of the TCP or port nonsense here; and finally
  5. Select the Source -- Initial Catalog to be [SQL Azure database you’ve already created on your service but haven’t necessarily set a schema for yet.]

Once you’ve cleared this hurtle it’s pretty much smooth sailing.

If you have any questions about this process or why I didn’t mention some of the available alternatives, go ahead and ask them in the comments and I’ll get back to you.

If you enjoyed this post, make sure you subscribe to my RSS feed!



Rise of the Popped Collar Programmer

November 19, 2011 10:09 by Aaronontheweb in General // Tags: , , // Comments (14)

I am frankly disturbed by a trend that I’ve seen both in-person and all over Hacker News / Reddit through the past year, and I am going to finally give it a name: “popped collar programming.”

Popped collar programming is the hipsterization of software development, and it’s happening in a co-working space, unprofitable venture-backed startup, or coffee shop near you.

A popped collar programmer’s life begins like this:

  1. [New Technology X] is released and offers a new and interesting view of how to perform [programming chore Y];

  2. Reddit / Hacker News announces the release of  [New Technology X] and is greeted with tons of enthusiasm and applause;

  3. Curious developers check [New Technology X] out – it’s an early release and thus they discover [New Technology X] is missing [Critical Features 1….N] and has [Stability Issues 1…N]; most of the early adopters utilize the technology sparingly in production and only where it’s the right solution for the problems they’re trying to solve.

  4. [Group of early adopters A] can’t get enough of [New Technology X] – they create lots of blog content on how to couple it alongside other popular technologies and receive front-page treatment on Reddit / Hacker News.

  5. [New Technology X], despite lacking [Critical Features 1….N]; having [Stability Issues 1…N]; and often not being the best business case match is now used in 100% of production projects by [fanboys of early adopters A].

  6. [Fanboys of early adopters A] declare the death of [Established Competitive Technology with Massive Following X]; [fanboys of early adopters A] have now become popped-collar programmers.

  7. [Popped Collar Programmers] begin purchase of ironic  / retro t-shirts; growing porn star mustaches;  writing blog entries about the challenges of scaling [New Technology X] despite having a trivial number of users on their service; blog entries make front page of Hacker News.

Popped Collar Programming is essentially “adopting technologies for the sake of appearances” – technologists who’ve fallen victim to this way of thinking often make technology decisions without any regard for what’s the best tool for the job. 

Popped Collars Bro

Once you’ve gone down the road of letting anonymous people on the internet convince you that having cool war stories with cutting edge technologies you can tell on Hacker News is more important than shipping software that works and is easy-to-maintain, you’ve lost the objectivity and pragmatism needed to be an effective software developer.


And it doesn’t stop at technology decisions; popped collar programmers spend more time picking out ironic t-shirts to go with their fedoras and oversize rubber watches than they do logging bugs.

Don’t Be That Guy

Popped Collar Programmers are inefficient (fitting square pegs in round holes from the start) and ineffective (building complex bug-prone architectures themselves to address problems that are already solved with widely-used technologies.)

If that argument’s not enough to convince you to stop yourself or your friends from being popped collar programmers, let’s try a live-by-peer-pressure, die-by-peer-pressure approach: you look like a complete pair of clownshoes in front of people who actually know what they’re doing.

If this sounds like you or someone you know, don’t be that guy. The solution here is to read The Pragmatic Programmer until you can’t stand to look at that Hadoop + Riak + Redis + Clojure-powered blog platform you created without vomiting in disgust.
Don’t. Be. That. Guy.

If you enjoyed this post, make sure you subscribe to my RSS feed!



How to Use Dependency Injection (Ninject) with WCF Services

August 16, 2011 15:32 by Aaronontheweb in .NET, WCF // Tags: , // Comments (1)

I spent a lot of time late last week trying to figure out exactly how to set up dependency injection for a WCF service I was developing on Windows Azure; there’s some documentation out there on how to do it, but it’s not nearly as helpful as I would like it. Thus I decided to document how I made WCF and Ninject play nice with each other, and I even provide a sample template you can use yourself.

Step 1 – Install Ninject.Extensions.WCF via NuGet

The developers who maintain Ninject were kind enough to put together a set of extensions for working with WCF applications specifically, and we can install this via NuGet (install it from CodePlex.)

Simply type this into the “package manager console” in Visual Studio: install-package Ninject.Extensions.Wcf

image

This will automatically grab the appropriate version of Ninject and the other binaries you’ll need to get the WCF ServiceHost to play nice with your service.

Step 2 – Write Your Own Ninject Module

The next step is to set up your dependency injections by writing your own NinjectModule like this:

using Ninject.Modules;

namespace wcf_ninject.DI
{
    public class WCFNinjectModule : NinjectModule
    {
        public override void Load()
        {
            //Injects the constructors of all DI-ed objects 
            //with a LinqToSQL implementation of IRepository
            Bind<IRepository>().To<LinqToSQLRepository>();
        }
    }
}

Step 3 – Add the “Factory” Attribute to Service1.svc

The next thing you need to do is tell the service host to use the Ninject Factory to instantiate the WCF service by way of a custom ServiceHostFactory included in the Ninject WCF Extensions.

To do this, right click on the “Service1.svc” file (or whatever you name your service) and select the “View Markup” option:

image

Once there, simply add this line:

Before

 

<%@ ServiceHost 
    Language="C#" 
    Debug="true" 
    Service="wcf_ninject.Service1" 
    CodeBehind="Service1.svc.cs" %>

After

<%@ ServiceHost 
    Language="C#" 
    Debug="true" 
    Service="wcf_ninject.Service1" 
    CodeBehind="Service1.svc.cs"
    Factory="Ninject.Extensions.Wcf.NinjectServiceHostFactory" %>

Step 4 – Add Global.asax Make It Inherit from NinjectWcfApplication

The final step is to add a Global.asax file to your WCF project and make it inherit from the NinjectWcfApplication class, like this:

public class Global : NinjectWcfApplication

In order for this code to compile, you’ll need to add a “CreateKernel” method to Global.asax – this method is what utilizes your Ninject module and uses it to inject dependencies into your WCF service. Here’s what that code looks like for this example:

protected override IKernel CreateKernel()
{
    return new StandardKernel(new WCFNinjectModule());
}

Now there’s just one last step…

Step 5 – Remove the Default Constructor from Your WCF Service; Have Dependency-Accepting Constructor Instead

Your final constructor in this example should look something like this:

 

 public Service1(IRepository repository)
{
    _repo = repository;
}

Get rid of any default constructors (ones that don’t accept arguments) – Ninject will raise an error if it finds one in the signature of your class as of writing this.

Grab the WCF + Ninject Visual Studio 2010 Project off of Github

You can download the Visual Studio 2010 project that contains this WCF + Ninject sample off of Github if you want to use it as a starting point.

If you enjoyed this post, make sure you subscribe to my RSS feed!



New Open Source Project: MVC.Utilities

August 14, 2011 07:33 by Aaronontheweb in ASP.NET, Open Source // Tags: // Comments (0)

I announced this on Twitter late last week, but I open-sourced a number of common helpers and service interfaces that I use throughout all of my production ASP.NET MVC applications and wrapped it into a project I call MVC.Utilities.

MVC.Utilities has a number of helper classes for the following:

  1. Encryption;
  2. Caching;
  3. Authentication;
  4. Routing;
  5. Security for user-uploaded files;
  6. and display helpers for things like Hacker News-style datetimes.

 

If you want more details and examples of what the library does exactly, be sure to check out the MVC.Utilities Wiki on Github. All of these services are meant to be Dependency-Injected into ASP.NET MVC controllers and are designed as such.

The library has a lot of room to expand and grow, and I encourage your ideas and contributions to the project! Just create a fork of MVC.Utilities on Github and send me a pull request; props to Wesley Tansey for already submitting a couple of patches!

Installing MVC.Utilities via NuGet

MVC.Utilities is available via NuGet – type in the following command via the Package Manager Console in Visual Studio to have it added to your project today:

Install-Package mvc-utilities

If you enjoyed this post, make sure you subscribe to my RSS feed!



How to Recruit a Technical Co-Founder for Your Startup

August 3, 2011 12:54 by Aaronontheweb in Startup // Tags: , // Comments (2)

The LA startup scene is fascinating, having lived and worked in it for a year now - it's a scene teeming with brillaint people with big ideas, and it's starting to attract some major capital from the Bay Area. It has one major issue: a big shortage of technical co-founders.

As a result, people like me get approached fairly regularly by companies that are either trying to recruit me or recruit through me - the vast majority of the time it's a pair of non-technical co-founders looking to third founder aboard, a technical cofounder, to build the MVP prior to raising some money. In my personal experience, the majority of pitches I've received have been poorly calculated and in need of much improvement.

Speaking as a former and future technical founder, I wanted to share my perspective on what non-technical (and technical, for that matter) founders could do differently to try to bring an early tech guy / gal onboard.

Don't Pitch Airtight Ideas; Start a Conversation

The last pitch most technical people entrepreneurial enough to leave a well-compensated job for a startup with no income want hear is one where a product-oriented founder presents an airtight idea that has no room for discussion whatsoever; every single pixel and user interaction has been planned, as has the business model and everything else. They just need a code monkey to build it.

If this sounds like you, start over or hire an outsourcing firm. Founders want to leave their mark on the the business itself and that means all aspects of it, whether it's the branding, the commercial model, or the product's implementation. Sure, a technical co-founder will defer to a commerce person on the specifics of the customer acquisition strategy or the product person on the details of the UX, but that doesn't mean that they want to sit out of those conversations in their entirety.

Any technical co-founder who's willing to take the risk to leave a high-paying job where they do someone else's bidding isn't going to take a not-yet-paying job to do someone else's bidding. They want intellectual co-ownership as much as any of the other founders.

A better way to do this is to start a conversation about a business idea in general, get the people you want as technical co-founders to buy-in and contribute their own ideas into the business, and let them take some degree of intellectual ownership over the project. Don't pitch! Ask questions - get the people you want to co-found with you involved. The amount of work someone like me will do without cashmoney for a project that we "co-own" is dramatically higher than if we were working on "someone else's project" under the same conditions. Read Leadership is an Art and remember what Max Depree says about letting your creative giants roam free; that applies here!

If you're not comfortable letting a geek (or anyone other than yourself) into the idea ownership fold, you're probably not mature enough to found a company.

Build a Demo Yourself

You know what impresses the hell out of me? When a non-technical person has the drive to build an early prototype or a set of wireframes themselves to help illustrate the concept and what they ultimately want to do.

There's still room in the business model and product idea for me to get some mindshare (see point #1,) but the fact that the person was passionate and humble enough to build a cludgy demo that doesn't work quite right tells me that they're not going to flake out and move onto something else while I'm left holding the bag.

A demo is also a great way to get a tech co-founder to buy in, see where improvements can be made, and help them understand the business.

Prove Your Worth

I know I can code, and I can prove it with my Github profile or my portfolio - how do I know that you can aquire customers or design a truly usuable prodct? Go out of your way to demonstrate that you've got skills - get some early partners signed up early; build a blog and a following around whatever idea you have; get some user testominals; do some UX testing; and so forth.

Present an Interesting Challenge

You know what intrigues me, as a developer? Solving big problems and helping a business grow. You know what doesn't? Modifying a WordPress blog for a lame-assed content play. Give me (and other technical types) something big we can chew on - we dig that. It's not that we're looking for overly-complicated solutions for simple problems - we're looking for interesting problems.

Here's how you can tell if your problem is interesting or not: is there an off-the-shelf product that does this? If the answer is "yes," you probably don't need a techincal co-founder.

If you need some advice, swing by Coloft and I'd be happy to talk with you. I'm usually there ;)

 

If you enjoyed this post, make sure you subscribe to my RSS feed!



MongoDB vs. SQL Server 2008: A .NET Developer’s Perspective

June 30, 2011 16:01 by Aaronontheweb in ASP.NET, MongoDB, SQL Server // Tags: // Comments (22)

One of the first projects I put together this year was Captain Obvious, a nifty little application that runs off of AppHarbor and ASP.NET MVC3. What made Captain Obvious special for me was that it was my first time using something other than a relational database1 in production – I chose MongoDB because it stands out to me as a lightweight, easy-to-work with store that’s easier to use for most CRUD applications. Since then I’ve gone on to build other projects which depend on Mongo.

What I’ve learned since is that MongoDB and SQL Server are tools that aren’t 100% interchangeable and are more situational than dogmatists make them out to be.

My goal in writing this is to help inform you on how you should decide to judge these two technologies as options for your ASP.NET / WebMatrix / WCF applications.

Relational Data Models vs. Document Models

The key to using Mongo or SQL Server effectively is understanding how the underlying data model works and how this impacts your ability to read / write what you want when you want to the datastore. Below are illustrations of the relational (SQL) model versus the document model.

mongo vs sql differences

 

In a relational model all of your information is expressed in the form of tables, all of which contain keys and some of which contain foreign keys to other tables. All of the information you read from and write to the database is expressed as either adding rows to these tables or combining their values based on some keys.

In a document model you have a relatively flat collection of items all identified by one primary key2, and instead of defining a relationship between two collections you simply embed one inside the other. So the relationship between the three tables in our relational model is expressed as a single, flat document in this model.

It is important to note here that there’s no schema that tells MongoDB what fields should or shouldn’t be expected in each document in the “Things” collection – in the relational universe each table is strongly, declaratively typed and every single item inserted into the row must conform to all of the constraints imposed by the relational database management system (RDBMS.) Anything goes in Mongo (although this can cause major problems, as we will see later.)

What Do Relational and Document Models Have in Common?

So what do these two data models have in common?

  1. Both support the notion of primary keys and indexes, and MongoDB can support multiple indices if needed;
  2. Both support queries and have models for sorting / limiting results;
  3. Both support the ability to reference other documents / tables (“wha? in Mongo? Yup.”)
  4. Both have a strong typing system; and
  5. Both support aggregation operations like SUM(), COUNT(), etc…

 

Seems pretty straightforward, right? But what’s up with documents being able to refer to each other?

As it turns out, implementing a database where every possible piece of information of interest to users is all embedded inside of its own distinct document comes with some drawbacks, so the creators of MongoDb added support for DbReference and the ability to do cross-collection references as well. A necessary evil in the name of practicality.

What’s Different between Relational and Document Models?

So what’s really different between the two models?

  1. Document models don’t have SQL – their query tools are extremely primitive in contrast (but the models are also much simpler and don't require sophisticated queries;)
  2. Fetching document references in the document model has to be done inside of separate queries3 whereas they can be done all in the same transaction in the relational model;
  3. Document models don’t have a schema – each document in a collection can have extra fields or fields with different type values, whereas all rows must conform to the same set of constraints in order to be inserted;
  4. In the document model types are associated with data upon assignment, rather than declared in advance;
  5. Document models have much more primitive tools for performing aggregations (edit: actually, not necessarily true for Mongo - it has built-in MapReduce which is powerful and sophisticated;) and
  6. Queries are defined on a per-collection basis in the document model, whereas a query in the relational model is an abstraction that simply refers to any number of related tables.

 

The picture that should be starting to form in your head of MongoDb vs. SQL Server at this point is a flat, simple system on one side and a multi-dimensional, rich system on the other. This is how I look at the two technologies in contrast to each other.

What’s Different about Developing .NET Apps Against Mongo and SQL Server 2008?

So at the end of the day, what's the bottom line for .NET developers who want to use Mongo or SQL Server in a web application? What are the REAL trade-offs?

Mongo IS the OR/M

The core strength of Mongo is in its document-view of data, and naturally this can be extended to a "POCO" view of data. Mongo clients like the NoRM Project in .NET will seem astonishingly similar to experienced Fluent NHibernate users, and this is no accident - your POCO data models are simply serialized to BSON and saved in Mongo 1:1. No mappings required.

I'm going to show you two similar pieces of source code I am using in production - one using NoRM and MongoDb on CaptainObvious and the other using Dapper and SQL Azure on XAPFest:

NoRM:

public IdeaResultSet GetIdeasByAuthor(int authorID, int offset = 0, int count = 10)
        {
            using (var db = Mongo.Create(ConnectionString))
            {
                var ideas =
                    db.GetCollection().AsQueryable().Where(x => x.AuthorReference.Id == authorID)
                    .Skip(offset)
                    .OrderByDescending(
                        x => x.DatePosted).Take(count).ToList();

                var totalIdeas = db.GetCollection().AsQueryable().Where(x => x.AuthorReference.Id == authorID).Count();

                //Fetch the referenced authors before we serve up the list again
                foreach (var idea in ideas)
                {
                    idea.Author = idea.AuthorReference.Fetch(() => db);
                }

                var resultSet = new IdeaResultSet { Ideas = ideas, 
                    PaginationValues = new PaginationTuple { 
                        MaxPages = PageCounterHelper.GetPageCount(totalIdeas, count), 
                        CurrentPage = PageCounterHelper.GetPageCount(offset, count) } 
                };

                return resultSet;
            }
        }

Dapper:

public IList<XfstApp> GetAppsByUser(string userName)
        {
            using (var conn = new SqlConnection(ConnectionString))
            {
                try
                {
                    conn.Open();

                    var appResults = conn
                        .Query(@"SELECT * FROM Apps
                                            INNER JOIN AppOwners ON AppOwners.AppName = Apps.AppName
                                            WHERE LOWER(AppOwners.UserName) = LOWER(@UserName)
                                            ORDER BY Apps.DateCreated ASC", new { UserName = userName });

                    //Return whatever we were able to collect
                    return appResults.Select(x => x.ToApp()).ToList();
                }
                catch (SqlException ex)
                {
                    TraceError(ex);
                    return new List<XfstApp>();
                }
                finally
                {
                    //Close the connection when we're finished, regardless of what happened
                    conn.Close();
                }
            }
        }

 

The amount of source code for these two technologies or the nature of it isn't wholly different.... What is drastically different is how I was thinking about the data when I was writing this code - I can save an instance of an Idea object to MongoDb on CaptainObvious without ever having created the collection first or defined a schema.

Whenever I want to look up an idea, I just pick one based off of a key value that I specify and I don't worry about any joins or anything (although I do have to load objects from the author collection if I need to display the author's name and contact info.)

In the SQL universe, I have to define my tables in advance and each time I want to extract an object, I have to think of it in terms of combined relationships between my tables and this requires a more thoughtful approach.

Mongo, in other words, lends itself to rapid application development whereas SQL Server has some innate friction built into any schema-based system.

In Mongo, the DBMS Isn't There to Protect Your Data's Integrity

One of the major advantages of a schema-based DBMS is that if the data a calling application tries to insert something that doesn't fit the schema into a row, the operation always fails. In Mongo, this isn't true - you can have one record in a collection with extra fields or fields of an odd type, and it can totally screw up the BSON serializer when it tries to process the collection (depending upon how flexible the serializer is.)

SQL users take this for granted, but when you have issues in Mongo along these lines they can be really frustrating to solve and difficult to debug.

In Mongo, Operations May Not Be Atomic

Operations are not atomic in Mongo by default, so all sorts of fun things can happen when you have multiple users changing properties on the same document. You can set an atomic flag to true, but even then operations still aren't really atomic (they're written from memory to disc in bulk.)

If you use Mongo and carry the same ACID assumptions that we learned on SQL Server, you might be in for a nasty surprise :p

Conclusion

Overall, the biggest difference between these two technologies is the model and how developers have to think about their data. Mongo is better suited to rapid application development, but in my opinion falls apart in scenarios where ACID-compliant systems are a must, like anything that goes anywhere near a financial transaction.

But, that's just my opinion :p


1typically I’ve only used SQL Server / MySQL in the past

2You can add indices to other fields in Mongo too, but that’s outside the scope of this article

3unless I am doing it wrong, which I may well be doing

If you enjoyed this post, make sure you subscribe to my RSS feed!



How to Securely Verify and Validate Image Uploads in ASP.NET and ASP.NET MVC

One of the more interesting things I had to do as part of building XAPFest was handle bulk image uploads for screenshots for applications and user / app icons. Most of the challenges here are UI-centric ones (which I resolved using jQuery File-Upload) but the one security challenge that remains outstanding is ensuring that the content uploaded to your servers is safe for your users to consume.

Fortunately this problem isn't too hard to solve and doesn't require much code in C#.

Flawed Approaches to Verifying Image Uploads

Here's what I usually see when developers try to allow only web-friendly image uploads:

  1. File extension validation (i.e. only allow images with .png, .jp[e]g, and .gif to be uploaded) and
  2. MIME type validation.

So what's wrong with these techniques? The issue is that both the file extension and MIME type can be spoofed, so there's no guarantee that a determined hacker might not take a js. file, slap an extra .png extension somewhere in the mix and spoof the MIME type.

Stronger Approach to Verifying Image Uploads: GDI+ Format Checking

Every file format has to follow a particular codec / byte order convention in order to be read and executed by software. This is as true for proprietary formats like .pptx as it is for .png and .gif.

You can use these codecs to your advantage and quickly tell if a file is really what it says it is - you quickly check the contents of the file against the supported formats' codecs to see if the content fits into any of those specifications.

Luckily GDI+ (System.Drawing.Imaging), the graphics engine which powers Windows, has some super-simple functions we can use to perform this validation. Here's a bit of source you can use to validate a file against PNG, JPEG, and GIF formats:

using System.Drawing.Imaging;
using System.IO;
using System.Drawing;

namespace XAPFest.Providers.Security
{
    /// 
    /// Utility class used to validate the contents of uploaded files
    /// 
    public static class FileUploadValidator
    {
        public static bool FileIsWebFriendlyImage(Stream stream)
        {
            try
            {
                //Read an image from the stream...
                var i = Image.FromStream(stream);

                //Move the pointer back to the beginning of the stream
                stream.Seek(0, SeekOrigin.Begin);

                if (ImageFormat.Jpeg.Equals(i.RawFormat))
                    return true;
                return ImageFormat.Png.Equals(i.RawFormat) 
|| ImageFormat.Gif.Equals(i.RawFormat);
            }
            catch
            {
                return false;
            }
        }

    }
}

All this code does is read the Stream object returned for each posted file into an Image object, and then verifies that the Image supports one of three supported codecs1.

This source code has not been tested by security experts, so use it at your own risk.

If you have any questions about how this code works or want to learn more, please drop me a line in the comments below or on Twitter.

Bonus: How Do I Make Sure Files Are below [X] Filesize?

Since I had this source code lying around anyway, I thought I would share it: 

public static bool FileIsWebFriendlyImage(Stream stream, long size)
        {
            return stream.Length <= size && FileIsWebFriendlyImage(stream);
        }
    }

Super-simple, like I said, but it gets the job done. Express the maximum allowable size as a long and compare it against the length of the stream

 


1The other important catch to note here is that I move the Stream's pointer back to the front of the stream, so it can be read again by the caller which passed the reference to this function.

If you enjoyed this post, make sure you subscribe to my RSS feed!



How I Built CaptainObvio.us

Captain Obvio.us - a place to share ideasI made a tiny splash on Hacker News a month ago when I asked for feedback on my newest side project, CaptainObvio.us – a simple portal for sharing ideas and soliciting feedback from a community of peers. The idea was popular and I’ve received a ton of feedback – I’ve implemented most of the Hacker News community’s suggestions but haven’t had the chance to do another round of customer development.

What I wanted to share in this blog post was some of the secret sauce I used for creating CaptainObvio.us – I originally created it mostly to learn MongoDB, and learned way more than that along the way.

Webstack: ASP.NET MVC3 on AppHarbor

I used ASP.NET MVC3 as my webstack of choice with AppHarbor as my hosting platform. ASP.NET MVC3 is a massive improvement over MVC2, and I took advantage of Razor syntax, the built-in support for DI (dependency injection) on controllers, and wrote a number of customized helpers to do things like create an action filter for Twitter @Anywhere.

AppHarbor has been a great experience to work with - I use Git for souce control for most of my personal projects like this one so deployments are a breeze on AppHarbor, but the other major reason I picked AppHarbor is that it shares the same Amazon AWS datacenter as MongoHQ - another [Thing]-as-a-Service that I used for hosting my MongoDB instance.

Data: MongoDB on MongoHQ

The original purpose of CaptainObvio.us was for me to learn MongoDB, a schemaless (NoSQL) document database written in C++ that is becoming all the rage in the Ruby on Rails universe. CaptainObvio.us is a good fit for Mongo given that the vast majority of its content consists of simple documents with a small amount of relational data for tying authors to ideas / comments and so forth.

I could not have gotten very far with MongoDB in C# were it not for the NoRM Project MongoDB drivers for C# - NoRM's drivers are much better than the default MongoDB drivers for C# and work similarly to LINQ-to-SQL (although not exactly.) It was a matter of hours for me to go from installing Mongo to having a functional site running with NoRM and ASP.NET MVC3.

Authentication: Originally Twitter @Anywhere; Sign-in-with-Twitter and Hammock Later

Twitter @Anywhere is a fairly new JavaScript-only framework for integrating Twitter into existing websites quickly and effortlessly - it automates all of the OAuth workflow, comes with tons of useful built-in widgets, and eliminates the need to write much (if any) plumbing needed to support Twitter integration.

This is all nice in theory but if you're building a site like CaptainObvious where your users use Twitter to sign-in and leave behind persistent data in your own data store, then this framework can really cause problems. I had to gut Twitter @Anywhere eventually because the post-authentication event hook would misfire on occasion due to issues on the client and thus I would have an "authorized" user running around adding comments and voting despite no record of them existing in the database.

In order to resolve the issue, I dumped Twitter @Anywhere and went with traditional Sign-in-with-Twitter powered by Hammock, which works fine.

Final Thoughts

I'm going to continue working on CaptainObvio.us, although I put it off largely due to all of the work I had to do engineering XAPFest's online presence on Windows Azure. If the project taught me anything, it's the value of continuous integration environments like AppHarbor and the ease with which you can get something up and running quickly with MongoDB.

I'd highly recommend checking out AppHarbor, MongoHQ, and the NoRM drivers if you're a .NET developer who's new to Mongo and wants to learn how to use it. I guarantee you that you'll appreciate traditional .NET databases like SQL Server 2008 and SQL Azure a lot better after you've worked with Mongo, as it'll help you learn the strengths and weakness of each respesctive platform.

If you enjoyed this post, make sure you subscribe to my RSS feed!



Search

About

My name is Aaron, I'm an entrepreneur and a .NET developer who develops web, cloud, and mobile applications.

I left Microsoft recently to start my own company, MarkedUp - we provide analytics for desktop developers, focusing initially on Windows 8 developers.

You can find me on Twitter or on Github!

Recent Comments

Comment RSS

Sign in