WebRequest Imitating Curl in C#

This is a quick tutorial on creating a WebRequest to imitate a Curl GET in C#. Curl is a fast and simple way to GET and post CSV data. I had previously looked at using IronPython, a python library for .net, in my project to directly call python, but the library doesn't support pycurl yet. Since Curl and Python aren't installed on Windows systems by default, I decided to use a WebRequest because it makes the code more portable, and doesn't require any extra installation steps on the server. You can still use this code on Linux and Mac systems, but that would require using Monocode.

After the WebRequest, there's also a short bit on usage where I will show you how to use the JavaScriptSerializer to deserialize the JSON string into a predefined object. This will enable you to use Linq on the object, and open up a wealth of uses of the data within your app. Limiting Curl requests for data in favor of building a data model will result in faster processing of the data.

getRecordsByTable returns a built JSON string to return the data from your supplied table name.

public class CurlTest
{
   public string getRecordsByTable(string tableName)
   {
               // build the request url with the tableName appended to the end.
               WebRequest request = WebRequest.Create("https://hostsite/" + tableName);

               // use the GET method.
               request.Method = "GET";

        // get this information from settings in your web config.
               string userName = "someone@somesite.something";
               string password = "password";

               string credentials = userName + ":" + password;
               request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(credentials));

        //  this is a hack to test ssl without a certificate.
               ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(AcceptAllCertifications);
   
   // create a web response
               WebResponse response = request.GetResponse();
   
   // create a data stream.
               Stream dataStream = response.GetResponseStream();
   
               // create a stream reader.
               StreamReader reader = new StreamReader(dataStream);
   
               // read the content into a string
               string serverResponse = reader.ReadToEnd();
   
   // map the CSV data to a JSON string.
               var mappedData = mapTableToJSON(serverResponse);
   
               // clean up.
              reader.Close();
              dataStream.Close();
              response.Close();

              return mappedData;
   }
}

The AcceptAllCertifications callback returns true to acknowledge that the host SSL certificate is valid. I wouldn't recommend this approach on production, but it's a great way to test a secure request over https. If you want to do a request over an unsecure site, just change https to http in your WebRequest url, and leave out the authentication parts of the code.

        public bool AcceptAllCertifications(object sender, System.Security.Cryptography.X509Certificates.X509Certificate certification, System.Security.Cryptography.X509Certificates.X509Chain chain, System.Net.Security.SslPolicyErrors sslPolicyErrors)
        {
            return true;
        }

The mapTableToJSON method takes the CSV string data, and formats that data into a JSON string.

public static string mapTableToJSON(string value)
        {
            // get lines out of the CSV data.
            if (value == null) return null;
            string[] lines = value.Split('\n');

            // get headers out of the CSV data.
            string[] headers = lines.First().Split(',');

            // build the JSON string.
            StringBuilder sb = new StringBuilder();
   // open the JSON formatting.
            sb.AppendLine("[");
   // iterate through the lines for fields.
            for (int i = 1; i < lines.Length; i++)
            {
    // regex selects comma delimiters that are not in quotes in order to avoid syntax issues because some fields may include a comma in the value.
                string[] fields = Regex.Split(lines[i], @",(?=(?:[^""]*""[^""]*"")*(?![^""]*""))");
                int a = 0;
                foreach (var f in fields)
                {
     // insert empty quotes as a JSON value for a header if the CSV value is empty.
                    if (String.IsNullOrEmpty(f))
                    {
                        fields[a] = "\"\"";
                    }
                    a++;
                }
    // format the headers and fields and output to an array.
                var elements = headers.Zip(fields, (header, field) => string.Format("{0}: {1}", header, field)).ToArray();
                string json = "{" + string.Format("{0}", string.Join(",", elements)) + "}";
    // apply commas after each json object except the last one to avoid syntax errors.
                if (i < lines.Length - 1)
                {
                    json += ",";
                }
    // append the JSON to the string builder.
                sb.AppendLine(json);
            }
   // close the JSON formatting.
            sb.AppendLine("]");

            return sb.ToString();
        }

Here is an example of usage, and an example of mapping the JSON string to our predefined Person object. The JSON header values must literally match your object variables.

public class Person
{
   public int id { get; set; }
   public string name { get; set; }
   public string comments { get; set; }
}

JSON string example: 
[
        {"id": "1", "name": "Darth Vader", "comments": "I find your lack of faith disturbing."},
        {"id": "2", "name": "Obi Wan Kenobi", "comments": "Sir Alec Guinness is awesome."},
        {"id": "3", "name": "Jar Jar Binks", "comments": "We can always delete him here."}
]

getAllPersonData is a method to show you how to map your JSON string to our predefined Person object.

public List<Person> getAllPersonData()
{
   // create an instance of the CurlTest class.
   CurlTest curlTest = new CurlTest();

   // use the getRecordsByTable method to get mapped JSON data.
   string jsonText = curlTest.getRecordsByTable("person");

   JavaScriptSerializer serializer = new JavaScriptSerializer();
   // if you have an issue with MaxJsonLength, make the length a higher number than the default.

   // deserialize the JSON object into a predefined object.
   List<Person> person = serializer.Deserialize<Person>(jsonText);

   //  :-)
   person.RemoveAll(a => a.name == "Jar Jar Binks");

   return person;
}

I will eventually include using parameters in the GET request, and show an example of posting CSV data.