F# Data


F# Data: HTTP Utilities

The .NET library provides a powerful API for creating and sending HTTP web requests. There is a simple WebClient type (see MSDN) and a more flexible HttpWebRequest type (see MSDN). However, these two types are quite difficult to use if you want to quickly run a simple HTTP request and specify parameters such as method, HTTP POST data, or additional headers.

The F# Data Library provides a simple Http type with four overloaded methods: RequestString and AsyncRequestString, that can be used to create a simple request and perform it synchronously or asynchronously, and Request and it's async companion AsyncRequest if you want to request binary files or you want to know more about the response like the status code, the response URL, or the returned headers and cookies.

To use the type, we first need to reference the library using #r (in an F# interactive) or add reference to a project. The type is located in FSharp.Data namespace:

1: 
2: 
#r "../../../bin/lib/net45/FSharp.Data.dll"
open FSharp.Data

Sending simple requests

To send a simple HTTP (GET) request that downloads a specified web page, you can use Http.RequestString and Http.AsyncRequestString with just a single parameter:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
// Download the content of a web site
Http.RequestString("http://tomasp.net")

// Download web site asynchronously
async { let! html = Http.AsyncRequestString("http://tomasp.net")
        printfn "%d" html.Length }
|> Async.Start

In the rest of the documentation, we focus on the RequestString method, because the use of AsyncRequestString is exactly the same.

Query parameters and headers

You can specify query parameters either by constructing an URL that includes the parameters (e.g. http://...?test=foo&more=bar) or you can pass them using the optional parameter query. The following example also explicitly specifies the GET method, but it will be set automatically for you if you omit it:

1: 
2: 
3: 
Http.RequestString
  ( "http://httpbin.org/get", 
    query=["test", "foo"], httpMethod="GET" )

Additional headers are specified similarly - using an optional parameter headers. The collection can contain custom headers, but also standard headers such as the Accept header (which has to be set using a specific property when using HttpWebRequest).

The following example uses The Movie Database API to search for the word "batman". To run the sample, you'll need to register and provide your API key:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
// API key for http://www.themoviedb.org
let apiKey = "<please register to get a key>"

// Run the HTTP web request
Http.RequestString
  ( "http://api.themoviedb.org/3/search/movie", httpMethod = "GET",
    query   = [ "api_key", apiKey; "query", "batman" ],
    headers = [ "Accept", "application/json" ])

The library supports a simple and unchecked string based API (used in the previous example), but you can also use pre-defined header names to avoid spelling mistakes. The named headers are available in HttpRequestHeaders (and HttpResponseHeaders) modules, so you can either use the full name HttpRequestHeaders.Accept, or open the module and use just the short name Accept as in the following example. Similarly, the HttpContentTypes enumeration provides well known content types:

1: 
open FSharp.Data.HttpRequestHeaders
1: 
2: 
3: 
4: 
5: 
// Run the HTTP web request
Http.RequestString
  ( "http://api.themoviedb.org/3/search/movie",
    query   = [ "api_key", apiKey; "query", "batman" ],
    headers = [ Accept HttpContentTypes.Json ])

Getting extra information

Note that in the previous snippet, if you don't specify a valid API key, you'll get a (401) Unathorized error, and that will throw an exception. Unlike when using WebRequest directly, the exception message will still include the response content, so it's easier to debug in F# interactive when the server returns extra info.

You can also opt out of the exception by specifying the silentHttpErrors parameter:

1: 
Http.RequestString("http://api.themoviedb.org/3/search/movie", silentHttpErrors = true)

This returns the following:

"{"status_code":7,"status_message":"Invalid API key: You must be granted a valid key.","success":false}"

In this case, you might want to look at the HTTP status code so you don't confuse an error message for an actual response. If you want to see more information about the response, including the status code, the response headers, the returned cookies, and the response url (which might be different to the url you passed when there are redirects), you can use the Request method instead of the RequestString method:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
let response = Http.Request("http://api.themoviedb.org/3/search/movie", silentHttpErrors = true)

// Examine information about the response
response.Headers
response.Cookies
response.ResponseUrl
response.StatusCode

Sending request data

If you want to create a POST request with HTTP POST data, you can specify the additional data in the body optional parameter. This parameter is of type HttpRequestBody, which is a discriminated union with three cases:

  • TextRequest for sending a string in the request body.
  • BinaryUpload for sending binary content in the request.
  • FormValues for sending a set of name-value pairs correspondent to form values.

If you specify a body, you do not need to set the httpMethod parameter, it will be set to Post automatically.

The following example uses the httpbin.org service which returns the request details:

1: 
Http.RequestString("http://httpbin.org/post", body = FormValues ["test", "foo"])

By default, the Content-Type header is set to text/plain, application/x-www-form-urlencoded, or application/octet-stream, depending on which kind of HttpRequestBody you specify, but you can change this behaviour by adding content-type to the list of headers using the optional argument headers:

1: 
2: 
3: 
4: 
Http.RequestString
  ( "http://httpbin.org/post", 
    headers = [ ContentType HttpContentTypes.Json ],
    body = TextRequest """ {"test": 42} """)

Maintaining cookies across requests

If you want to maintain cookies between requests, you can specify the cookieContainer parameter. The following example will request the MSDN documentation for the HttpRequest class. It will return the code snippets in C# and not F#:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
// Build URL with documentation for a given class
let msdnUrl className = 
  let root = "http://msdn.microsoft.com"
  sprintf "%s/en-gb/library/%s.aspx" root className

// Get the page and search for F# code
let docInCSharp = Http.RequestString(msdnUrl "system.web.httprequest")
docInCSharp.Contains "<a>F#</a>"

If we go to another MSDN page and click on a F# code sample, and then go back to the HttpRequest class documentation, while maintaining the same cookieContainer, we will be presented with the F# code snippets:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
open System.Net
let cc = CookieContainer()

// Send a request to switch the language
Http.RequestString
  ( msdnUrl "system.datetime", 
    query = ["cs-save-lang", "1"; "cs-lang","fsharp"], 
    cookieContainer = cc) |> ignore

// Request the documentation again & search for F#
let docInFSharp = 
  Http.RequestString
    ( msdnUrl "system.web.httprequest", 
      cookieContainer = cc )
docInFSharp.Contains "<a>F#</a>"

Requesting binary data

The RequestString method will always return the response as a string, but if you use the Request method, it will return a HttpResponseBody.Text or a HttpResponseBody.Binary depending on the response content-type header:

1: 
2: 
3: 
4: 
5: 
6: 
let logoUrl = "https://raw.github.com/fsharp/FSharp.Data/master/misc/logo.png"
match Http.Request(logoUrl).Body with
| Text text -> 
    printfn "Got text content: %s" text
| Binary bytes -> 
    printfn "Got %d bytes of binary content" bytes.Length

Customizing the HTTP request

For the cases where you need something not natively provided by the library, you can use the customizeHttpRequest parameter, which expects a function that transforms an HttpWebRequest.

As an example, let's say you want to add a client certificate to your request. To do that, you need to open the X509Certificates namespace from System.Security.Cryptography, create a X509ClientCertificate2 value, and add it to the ClientCertificates list of the request.

Assuming the certificate is stored in myCertificate.pfx:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
open System.Security.Cryptography.X509Certificates

// Load the certificate from local file
let clientCert = 
  new X509Certificate2(".\myCertificate.pfx", "password")

// Send the request with certificate
Http.Request
  ( "http://yourprotectedresouce.com/data",
    customizeHttpRequest = fun req -> 
        req.ClientCertificates.Add(clientCert) |> ignore; req)

Handling multipart form data

You can also send http multipart form data via the Multipart HttpRequestBody case. Data sent in this way is streamed instead of being read into memory in its entirety, allowing for uploads of arbitrary size.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
let largeFilePath = "//path/to/large/file.mp4"
let data = System.IO.File.OpenRead(largeFilePath) :> System.IO.Stream

Http.Request
  ( "http://endpoint/for/multipart/data", 
  body = Multipart(
    boundary = "define a custom boundary here", // this is used to separate the items you're streaming
    parts = [
      MultipartItem("formFieldName", System.IO.Path.GetFileName(largeFilePath), data)
    ]
  ))

Related articles

Multiple items
namespace FSharp

--------------------
namespace Microsoft.FSharp
Multiple items
namespace FSharp.Data

--------------------
namespace Microsoft.FSharp.Data
Multiple items
module Http

--------------------
type Http =
  private new : unit -> Http
  static member private AppendQueryToUrl : url:string * query:(string * string) list -> string
  static member AsyncRequest : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<HttpResponse>
  static member AsyncRequestStream : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<HttpResponseWithStream>
  static member AsyncRequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<string>
  static member private EncodeFormData : query:string -> string
  static member private EncodeUrlParam : param:string -> string
  static member private InnerRequest : url:string * toHttpResponse:(string -> int -> string -> string -> 'a0 option -> Map<string,string> -> Map<string,string> -> Stream -> Async<'a1>) * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:'a0 * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<'a1>
  static member Request : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> HttpResponse
  static member RequestStream : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> HttpResponseWithStream
  ...
static member Http.RequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:System.Net.CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(System.Net.HttpWebRequest -> System.Net.HttpWebRequest) * ?timeout:int -> string
val async : AsyncBuilder
val html : string
static member Http.AsyncRequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:System.Net.CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(System.Net.HttpWebRequest -> System.Net.HttpWebRequest) * ?timeout:int -> Async<string>
val printfn : format:Printf.TextWriterFormat<'T> -> 'T
property System.String.Length: int
Multiple items
type Async =
  static member AsBeginEnd : computation:('Arg -> Async<'T>) -> ('Arg * AsyncCallback * obj -> IAsyncResult) * (IAsyncResult -> 'T) * (IAsyncResult -> unit)
  static member AwaitEvent : event:IEvent<'Del,'T> * ?cancelAction:(unit -> unit) -> Async<'T> (requires delegate and 'Del :> Delegate)
  static member AwaitIAsyncResult : iar:IAsyncResult * ?millisecondsTimeout:int -> Async<bool>
  static member AwaitTask : task:Task -> Async<unit>
  static member AwaitTask : task:Task<'T> -> Async<'T>
  static member AwaitWaitHandle : waitHandle:WaitHandle * ?millisecondsTimeout:int -> Async<bool>
  static member CancelDefaultToken : unit -> unit
  static member Catch : computation:Async<'T> -> Async<Choice<'T,exn>>
  static member Choice : computations:seq<Async<'T option>> -> Async<'T option>
  static member FromBeginEnd : beginAction:(AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
  ...

--------------------
type Async<'T> =
static member Async.Start : computation:Async<unit> * ?cancellationToken:System.Threading.CancellationToken -> unit
val query : Linq.QueryBuilder
val apiKey : string
module HttpRequestHeaders

from FSharp.Data
val Accept : contentType:string -> string * string
module HttpContentTypes

from FSharp.Data
val Json : string
val response : HttpResponse
static member Http.Request : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:System.Net.CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(System.Net.HttpWebRequest -> System.Net.HttpWebRequest) * ?timeout:int -> HttpResponse
HttpResponse.Headers: Map<string,string>
HttpResponse.Cookies: Map<string,string>
HttpResponse.ResponseUrl: string
HttpResponse.StatusCode: int
union case HttpRequestBody.FormValues: seq<string * string> -> HttpRequestBody
val ContentType : contentType:string -> string * string
union case HttpRequestBody.TextRequest: string -> HttpRequestBody
val msdnUrl : className:string -> string
val className : string
val root : string
val sprintf : format:Printf.StringFormat<'T> -> 'T
val docInCSharp : string
System.String.Contains(value: string) : bool
namespace System
namespace System.Net
val cc : CookieContainer
Multiple items
type CookieContainer =
  new : unit -> CookieContainer + 2 overloads
  member Add : cookie:Cookie -> unit + 3 overloads
  member Capacity : int with get, set
  member Count : int
  member GetCookieHeader : uri:Uri -> string
  member GetCookies : uri:Uri -> CookieCollection
  member MaxCookieSize : int with get, set
  member PerDomainCapacity : int with get, set
  member SetCookies : uri:Uri * cookieHeader:string -> unit
  static val DefaultCookieLimit : int
  ...

--------------------
CookieContainer() : CookieContainer
CookieContainer(capacity: int) : CookieContainer
CookieContainer(capacity: int, perDomainCapacity: int, maxCookieSize: int) : CookieContainer
Multiple items
namespace System.Net.Http

--------------------
module Http

--------------------
type Http =
  private new : unit -> Http
  static member private AppendQueryToUrl : url:string * query:(string * string) list -> string
  static member AsyncRequest : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<HttpResponse>
  static member AsyncRequestStream : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<HttpResponseWithStream>
  static member AsyncRequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<string>
  static member private EncodeFormData : query:string -> string
  static member private EncodeUrlParam : param:string -> string
  static member private InnerRequest : url:string * toHttpResponse:(string -> int -> string -> string -> 'a0 option -> Map<string,string> -> Map<string,string> -> Stream -> Async<'a1>) * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:'a0 * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<'a1>
  static member Request : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> HttpResponse
  static member RequestStream : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> HttpResponseWithStream
  ...
static member Http.RequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> string
val ignore : value:'T -> unit
val docInFSharp : string
val logoUrl : string
static member Http.Request : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?silentCookieErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> HttpResponse
union case HttpResponseBody.Text: string -> HttpResponseBody
val text : string
union case HttpResponseBody.Binary: byte [] -> HttpResponseBody
val bytes : byte []
property System.Array.Length: int
namespace System.Security
namespace System.Security.Cryptography
namespace System.Security.Cryptography.X509Certificates
val clientCert : X509Certificate2
Multiple items
type X509Certificate2 =
  inherit X509Certificate
  new : unit -> X509Certificate2 + 12 overloads
  member Archived : bool with get, set
  member Extensions : X509ExtensionCollection
  member FriendlyName : string with get, set
  member GetNameInfo : nameType:X509NameType * forIssuer:bool -> string
  member HasPrivateKey : bool
  member Import : rawData:byte[] -> unit + 5 overloads
  member IssuerName : X500DistinguishedName
  member NotAfter : DateTime
  member NotBefore : DateTime
  ...

--------------------
X509Certificate2() : X509Certificate2
   (+0 other overloads)
X509Certificate2(rawData: byte []) : X509Certificate2
   (+0 other overloads)
X509Certificate2(fileName: string) : X509Certificate2
   (+0 other overloads)
X509Certificate2(handle: nativeint) : X509Certificate2
   (+0 other overloads)
X509Certificate2(certificate: X509Certificate) : X509Certificate2
   (+0 other overloads)
X509Certificate2(rawData: byte [], password: string) : X509Certificate2
   (+0 other overloads)
X509Certificate2(rawData: byte [], password: System.Security.SecureString) : X509Certificate2
   (+0 other overloads)
X509Certificate2(fileName: string, password: string) : X509Certificate2
   (+0 other overloads)
X509Certificate2(fileName: string, password: System.Security.SecureString) : X509Certificate2
   (+0 other overloads)
X509Certificate2(rawData: byte [], password: string, keyStorageFlags: X509KeyStorageFlags) : X509Certificate2
   (+0 other overloads)
val req : HttpWebRequest
property HttpWebRequest.ClientCertificates: X509CertificateCollection
X509CertificateCollection.Add(value: X509Certificate) : int
val largeFilePath : string
val data : System.IO.Stream
namespace System.IO
type File =
  static member AppendAllLines : path:string * contents:IEnumerable<string> -> unit + 1 overload
  static member AppendAllText : path:string * contents:string -> unit + 1 overload
  static member AppendText : path:string -> StreamWriter
  static member Copy : sourceFileName:string * destFileName:string -> unit + 1 overload
  static member Create : path:string -> FileStream + 3 overloads
  static member CreateText : path:string -> StreamWriter
  static member Decrypt : path:string -> unit
  static member Delete : path:string -> unit
  static member Encrypt : path:string -> unit
  static member Exists : path:string -> bool
  ...
System.IO.File.OpenRead(path: string) : System.IO.FileStream
type Stream =
  inherit MarshalByRefObject
  member BeginRead : buffer:byte[] * offset:int * count:int * callback:AsyncCallback * state:obj -> IAsyncResult
  member BeginWrite : buffer:byte[] * offset:int * count:int * callback:AsyncCallback * state:obj -> IAsyncResult
  member CanRead : bool
  member CanSeek : bool
  member CanTimeout : bool
  member CanWrite : bool
  member Close : unit -> unit
  member CopyTo : destination:Stream -> unit + 1 overload
  member CopyToAsync : destination:Stream -> Task + 2 overloads
  member Dispose : unit -> unit
  ...
union case HttpRequestBody.Multipart: boundary: string * parts: seq<MultipartItem> -> HttpRequestBody
Multiple items
union case MultipartItem.MultipartItem: formField: string * filename: string * content: System.IO.Stream -> MultipartItem

--------------------
type MultipartItem = | MultipartItem of formField: string * filename: string * content: Stream
type Path =
  static val DirectorySeparatorChar : char
  static val AltDirectorySeparatorChar : char
  static val VolumeSeparatorChar : char
  static val InvalidPathChars : char[]
  static val PathSeparator : char
  static member ChangeExtension : path:string * extension:string -> string
  static member Combine : [<ParamArray>] paths:string[] -> string + 3 overloads
  static member GetDirectoryName : path:string -> string
  static member GetExtension : path:string -> string
  static member GetFileName : path:string -> string
  ...
System.IO.Path.GetFileName(path: string) : string
Fork me on GitHub