F# Data


F# Data: HTTP Utilities

The .NET library provides a powerful API for creating and sending HTTP web requests. There is a simple WebClient type (see MSDN) and a more flexible HttpWebRequest type (see MSDN). However, these two types are quite difficult to use if you want to quickly run a simple HTTP request and specify parameters such as method, HTTP POST data, or additional headers.

The F# Data Library provides a simple Http type with four overloaded methods: RequestString and AsyncRequestString, that can be used to create a simple request and perform it synchronously or asynchronously, and Request and it's async companion AsyncRequest if you want to request binary files or you want to know more about the response like the status code, the response URL, or the returned headers and cookies.

To use the type, we first need to reference the library using #r (in an F# interactive) or add reference to a project. The type is located in FSharp.Net namespace:

1: 
2: 
#r "../../../bin/FSharp.Data.dll"
open FSharp.Data

Sending simple requests

To send a simple HTTP (GET) request that downloads a specified web page, you can use Http.RequestString and Http.AsyncRequestString with just a single parameter:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
// Download the content of a web site
Http.RequestString("http://tomasp.net")

// Download web site asynchronously
async { let! html = Http.AsyncRequestString("http://tomasp.net")
        printfn "%d" html.Length }
|> Async.Start

In the rest of the documentation, we focus on the RequestString method, because the use of AsyncRequestString is exactly the same.

Query parameters and headers

You can specify query parameters either by constructing an URL that includes the parameters (e.g. http://...?test=foo&more=bar) or you can pass them using the optional parameter query. The following example also explicitly specifies the GET method, but it will be set automatically for you if you omit it:

1: 
2: 
3: 
Http.RequestString
  ( "http://httpbin.org/get", 
    query=["test", "foo"], httpMethod="GET" )

Additional headers are specified similarly - using an optional parameter headers. The collection can contain custom headers, but also standard headers such as the Accept header (which has to be set using a specific property when using HttpWebRequest).

The following example uses The Movie Database API to search for the word "batman". To run the sample, you'll need to register and provide your API key:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
// API key for http://www.themoviedb.org
let apiKey = "<please register to get a key>"

// Run the HTTP web request
Http.RequestString
  ( "http://api.themoviedb.org/3/search/movie", httpMethod = "GET",
    query   = [ "api_key", apiKey; "query", "batman" ],
    headers = [ "Accept", "application/json" ])

The library supports a simple and unchecked string based API (used in the previous example), but you can also use pre-defined header names to avoid spelling mistakes. The named headers are available in HttpRequestHeaders (and HttpResponseHeaders) modules, so you can either use the full name HttpRequestHeaders.Accept, or open the module and use just the short name Accept as in the following example. Similarly, the HttpContentTypes enumeration provides well known content types:

1: 
open FSharp.Data.HttpRequestHeaders
1: 
2: 
3: 
4: 
5: 
// Run the HTTP web request
Http.RequestString
  ( "http://api.themoviedb.org/3/search/movie",
    query   = [ "api_key", apiKey; "query", "batman" ],
    headers = [ Accept HttpContentTypes.Json ])

Getting extra information

Note that in the previous snippet, if you don't specify a valid API key, you'll get a (401) Unathorized error, and that will throw an exception. Unlike when using WebRequest directly, the exception message will still include the response content, so it's easier to debug in F# interactive when the server returns extra info.

You can also opt out of the exception by specifying the silentHttpErrors parameter:

1: 
Http.RequestString("http://api.themoviedb.org/3/search/movie", silentHttpErrors = true)

This returns the following:

"{"status_message":"Invalid API key: You must be granted a valid key.","status_code":7}"

In this case, you might want to look at the HTTP status code so you don't confuse an error message for an actual response. If you want to see more information about the response, including the status code, the response headers, the returned cookies, and the response url (which might be different to the url you passed when there are redirects), you can use the Request method instead of the RequestString method:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
let response = Http.Request("http://api.themoviedb.org/3/search/movie", silentHttpErrors = true)

// Examine information about the response
response.Headers
response.Cookies
response.ResponseUrl
response.StatusCode

Sending request data

If you want to create a POST request with HTTP POST data, you can specify the additional data in the body optional parameter. This parameter is of type HttpRequestBody, which is a discriminated union with three cases:

  • TextRequest for sending a string in the request body.
  • BinaryUpload for sending binary content in the request.
  • FormValues for sending a set of name-value pairs correspondent to form values.

If you specify a body, you do not need to set the httpMethod parameter, it will be set to Post automatically.

The following example uses the httpbin.org service which returns the request details:

1: 
Http.RequestString("http://httpbin.org/post", body = FormValues ["test", "foo"])

By default, the Content-Type header is set to text/plain, application/x-www-form-urlencoded, or application/octet-stream, depending on which kind of HttpRequestBody you specify, but you can change this behaviour by adding content-type to the list of headers using the optional argument headers:

1: 
2: 
3: 
4: 
Http.RequestString
  ( "http://httpbin.org/post", 
    headers = [ ContentType HttpContentTypes.Json ],
    body = TextRequest """ {"test": 42} """)

Maintaining cookies across requests

If you want to maintain cookies between requests, you can specify the cookieContainer parameter. The following example will request the MSDN documentation for the HttpRequest class. It will return the code snippets in C# and not F#:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
// Build URL with documentation for a given class
let msdnUrl className = 
  let root = "http://msdn.microsoft.com"
  sprintf "%s/en-gb/library/%s.aspx" root className

// Get the page and search for F# code
let docInCSharp = Http.RequestString(msdnUrl "system.web.httprequest")
docInCSharp.Contains "<a>F#</a>"

If we go to another MSDN page and click on a F# code sample, and then go back to the HttpRequest class documentation, while maintaining the same cookieContainer, we will be presented with the F# code snippets:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
open System.Net
let cc = CookieContainer()

// Send a request to switch the language
Http.RequestString
  ( msdnUrl "system.datetime", 
    query = ["cs-save-lang", "1"; "cs-lang","fsharp"], 
    cookieContainer = cc) |> ignore

// Request the documentation again & search for F#
let docInFSharp = 
  Http.RequestString
    ( msdnUrl "system.web.httprequest", 
      cookieContainer = cc )
docInFSharp.Contains "<a>F#</a>"

Requesting binary data

The RequestString method will always return the response as a string, but if you use the Request method, it will return a HttpResponseBody.Text or a HttpResponseBody.Binary depending on the response content-type header:

1: 
2: 
3: 
4: 
5: 
6: 
let logoUrl = "https://raw.github.com/fsharp/FSharp.Data/master/misc/logo.png"
match Http.Request(logoUrl).Body with
| Text text -> 
    printfn "Got text content: %s" text
| Binary bytes -> 
    printfn "Got %d bytes of binary content" bytes.Length

Customizing the HTTP request

For the cases where you need something not natively provided by the library, you can use the customizeHttpRequest parameter, which expects a function that transforms an HttpWebRequest.

As an example, let's say you want to add a client certificate to your request. To do that, you need to open the X509Certificates namespace from System.Security.Cryptography, create a X509ClientCertificate2 value, and add it to the ClientCertificates list of the request.

Assuming the certificate is stored in myCertificate.pfx:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
open System.Security.Cryptography.X509Certificates

// Load the certificate from local file
let clientCert = 
  new X509Certificate2(".\myCertificate.pfx", "password")

// Send the request with certificate
Http.Request
  ( "http://yourprotectedresouce.com/data",
    customizeHttpRequest = fun req -> 
        req.ClientCertificates.Add(clientCert) |> ignore; req)

Related articles

Multiple items
namespace FSharp

--------------------
namespace Microsoft.FSharp
Multiple items
namespace FSharp.Data

--------------------
namespace Microsoft.FSharp.Data
type Http =
  private new : unit -> Http
  static member private AppendQueryToUrl : url:string * query:(string * string) list -> string
  static member AsyncRequest : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<HttpResponse>
  static member AsyncRequestStream : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<HttpResponseWithStream>
  static member AsyncRequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<string>
  static member private InnerRequest : url:string * toHttpResponse:(string -> int -> string -> string -> string -> 'a0 option -> Map<string,string> -> Map<string,string> -> Stream -> Async<'a1>) * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:'a0 * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> Async<'a1>
  static member Request : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> HttpResponse
  static member RequestStream : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> HttpResponseWithStream
  static member RequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> string

Full name: FSharp.Data.Http
static member Http.RequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:System.Net.CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(System.Net.HttpWebRequest -> System.Net.HttpWebRequest) * ?timeout:int -> string
val async : AsyncBuilder

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.async
val html : string
static member Http.AsyncRequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:System.Net.CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(System.Net.HttpWebRequest -> System.Net.HttpWebRequest) * ?timeout:int -> Async<string>
val printfn : format:Printf.TextWriterFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.printfn
property System.String.Length: int
Multiple items
type Async
static member AsBeginEnd : computation:('Arg -> Async<'T>) -> ('Arg * AsyncCallback * obj -> IAsyncResult) * (IAsyncResult -> 'T) * (IAsyncResult -> unit)
static member AwaitEvent : event:IEvent<'Del,'T> * ?cancelAction:(unit -> unit) -> Async<'T> (requires delegate and 'Del :> Delegate)
static member AwaitIAsyncResult : iar:IAsyncResult * ?millisecondsTimeout:int -> Async<bool>
static member AwaitTask : task:Task -> Async<unit>
static member AwaitTask : task:Task<'T> -> Async<'T>
static member AwaitWaitHandle : waitHandle:WaitHandle * ?millisecondsTimeout:int -> Async<bool>
static member CancelDefaultToken : unit -> unit
static member Catch : computation:Async<'T> -> Async<Choice<'T,exn>>
static member FromBeginEnd : beginAction:(AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
static member FromBeginEnd : arg:'Arg1 * beginAction:('Arg1 * AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
static member FromBeginEnd : arg1:'Arg1 * arg2:'Arg2 * beginAction:('Arg1 * 'Arg2 * AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
static member FromBeginEnd : arg1:'Arg1 * arg2:'Arg2 * arg3:'Arg3 * beginAction:('Arg1 * 'Arg2 * 'Arg3 * AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
static member FromContinuations : callback:(('T -> unit) * (exn -> unit) * (OperationCanceledException -> unit) -> unit) -> Async<'T>
static member Ignore : computation:Async<'T> -> Async<unit>
static member OnCancel : interruption:(unit -> unit) -> Async<IDisposable>
static member Parallel : computations:seq<Async<'T>> -> Async<'T []>
static member RunSynchronously : computation:Async<'T> * ?timeout:int * ?cancellationToken:CancellationToken -> 'T
static member Sleep : millisecondsDueTime:int -> Async<unit>
static member Start : computation:Async<unit> * ?cancellationToken:CancellationToken -> unit
static member StartAsTask : computation:Async<'T> * ?taskCreationOptions:TaskCreationOptions * ?cancellationToken:CancellationToken -> Task<'T>
static member StartChild : computation:Async<'T> * ?millisecondsTimeout:int -> Async<Async<'T>>
static member StartChildAsTask : computation:Async<'T> * ?taskCreationOptions:TaskCreationOptions -> Async<Task<'T>>
static member StartImmediate : computation:Async<unit> * ?cancellationToken:CancellationToken -> unit
static member StartWithContinuations : computation:Async<'T> * continuation:('T -> unit) * exceptionContinuation:(exn -> unit) * cancellationContinuation:(OperationCanceledException -> unit) * ?cancellationToken:CancellationToken -> unit
static member SwitchToContext : syncContext:SynchronizationContext -> Async<unit>
static member SwitchToNewThread : unit -> Async<unit>
static member SwitchToThreadPool : unit -> Async<unit>
static member TryCancelled : computation:Async<'T> * compensation:(OperationCanceledException -> unit) -> Async<'T>
static member CancellationToken : Async<CancellationToken>
static member DefaultCancellationToken : CancellationToken

Full name: Microsoft.FSharp.Control.Async

--------------------
type Async<'T>

Full name: Microsoft.FSharp.Control.Async<_>
static member Async.Start : computation:Async<unit> * ?cancellationToken:System.Threading.CancellationToken -> unit
val query : Linq.QueryBuilder

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.query
val apiKey : string

Full name: Http.apiKey
module HttpRequestHeaders

from FSharp.Data
val Accept : contentType:string -> string * string

Full name: FSharp.Data.HttpRequestHeaders.Accept
module HttpContentTypes

from FSharp.Data
val Json : string

Full name: FSharp.Data.HttpContentTypes.Json
val response : HttpResponse

Full name: Http.response
static member Http.Request : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:System.Net.CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(System.Net.HttpWebRequest -> System.Net.HttpWebRequest) * ?timeout:int -> HttpResponse
HttpResponse.Headers: Map<string,string>
HttpResponse.Cookies: Map<string,string>
HttpResponse.ResponseUrl: string
HttpResponse.StatusCode: int
union case HttpRequestBody.FormValues: seq<string * string> -> HttpRequestBody
val ContentType : contentType:string -> string * string

Full name: FSharp.Data.HttpRequestHeaders.ContentType
union case HttpRequestBody.TextRequest: string -> HttpRequestBody
val msdnUrl : className:string -> string

Full name: Http.msdnUrl
val className : string
val root : string
val sprintf : format:Printf.StringFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.sprintf
val docInCSharp : string

Full name: Http.docInCSharp
System.String.Contains(value: string) : bool
namespace System
namespace System.Net
val cc : CookieContainer

Full name: Http.cc
Multiple items
type CookieContainer =
  new : unit -> CookieContainer + 2 overloads
  member Add : cookie:Cookie -> unit + 3 overloads
  member Capacity : int with get, set
  member Count : int
  member GetCookieHeader : uri:Uri -> string
  member GetCookies : uri:Uri -> CookieCollection
  member MaxCookieSize : int with get, set
  member PerDomainCapacity : int with get, set
  member SetCookies : uri:Uri * cookieHeader:string -> unit
  static val DefaultCookieLimit : int
  ...

Full name: System.Net.CookieContainer

--------------------
CookieContainer() : unit
CookieContainer(capacity: int) : unit
CookieContainer(capacity: int, perDomainCapacity: int, maxCookieSize: int) : unit
static member Http.RequestString : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> string
val ignore : value:'T -> unit

Full name: Microsoft.FSharp.Core.Operators.ignore
val docInFSharp : string

Full name: Http.docInFSharp
val logoUrl : string

Full name: Http.logoUrl
static member Http.Request : url:string * ?query:(string * string) list * ?headers:seq<string * string> * ?httpMethod:string * ?body:HttpRequestBody * ?cookies:seq<string * string> * ?cookieContainer:CookieContainer * ?silentHttpErrors:bool * ?responseEncodingOverride:string * ?customizeHttpRequest:(HttpWebRequest -> HttpWebRequest) * ?timeout:int -> HttpResponse
union case HttpResponseBody.Text: string -> HttpResponseBody
val text : string
union case HttpResponseBody.Binary: byte [] -> HttpResponseBody
val bytes : byte []
property System.Array.Length: int
namespace System.Security
namespace System.Security.Cryptography
namespace System.Security.Cryptography.X509Certificates
val clientCert : X509Certificate2

Full name: Http.clientCert
Multiple items
type X509Certificate2 =
  inherit X509Certificate
  new : unit -> X509Certificate2 + 12 overloads
  member Archived : bool with get, set
  member Extensions : X509ExtensionCollection
  member FriendlyName : string with get, set
  member GetNameInfo : nameType:X509NameType * forIssuer:bool -> string
  member HasPrivateKey : bool
  member Import : rawData:byte[] -> unit + 5 overloads
  member IssuerName : X500DistinguishedName
  member NotAfter : DateTime
  member NotBefore : DateTime
  ...

Full name: System.Security.Cryptography.X509Certificates.X509Certificate2

--------------------
X509Certificate2() : unit
   (+0 other overloads)
X509Certificate2(rawData: byte []) : unit
   (+0 other overloads)
X509Certificate2(fileName: string) : unit
   (+0 other overloads)
X509Certificate2(handle: nativeint) : unit
   (+0 other overloads)
X509Certificate2(certificate: X509Certificate) : unit
   (+0 other overloads)
X509Certificate2(rawData: byte [], password: string) : unit
   (+0 other overloads)
X509Certificate2(rawData: byte [], password: System.Security.SecureString) : unit
   (+0 other overloads)
X509Certificate2(fileName: string, password: string) : unit
   (+0 other overloads)
X509Certificate2(fileName: string, password: System.Security.SecureString) : unit
   (+0 other overloads)
X509Certificate2(rawData: byte [], password: string, keyStorageFlags: X509KeyStorageFlags) : unit
   (+0 other overloads)
val req : HttpWebRequest
property HttpWebRequest.ClientCertificates: X509CertificateCollection
X509CertificateCollection.Add(value: X509Certificate) : int
Fork me on GitHub