Databases
StackOverflow2013
Postsdbo
PODeserializing twitter stream json string F#, how to define 'null' as a proper value?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PODeserializing twitter stream json string F#, how to define 'null' as a proper value?
    text
    copied!<p>I leave the below in case anybody wants the record types, but my question can be asked as following. Suppose one needs to write to a file a record type that can be null, what is the best way to achieve that, considering the below yields: "The type 'Basic' does not have 'null' as a proper value":</p> <pre><code>type Basic = {Toto:string} let basicToString b = function |null -&gt; "null" |record-&gt;record.Toto </code></pre> <p>Thanks a lot </p> <hr> <p>I have the following code to deserialize tweets from twitter stream:</p> <pre><code>open System.Runtime.Serialization open System.Runtime.Serialization.Json open System open System.Text open System.IO let makeString numFields = if numFields&gt;1 then seq[for i in 1..(numFields-1)-&gt;","] |&gt; Seq.fold(fun elem res-&gt; elem+res) "" else "" [&lt;DataContract&gt;] type boundingbox = { [&lt;field: DataMember(Name = "type")&gt;] t:string [&lt;field: DataMember(Name = "coordinates")&gt;] coordinates:float array array array } with member this.ToCsv() = let temp = string this if temp ="" then makeString 2 else this.t+","+"coordinates" [&lt;DataContract&gt;] type attributes = { [&lt;field: DataMember(Name = "street_adress")&gt;] street_adress:string } with member this.ToCsv() = let temp = string this if temp ="" then makeString 1 else this.street_adress [&lt;DataContract&gt;] type place = { [&lt;field: DataMember(Name = "country")&gt;] country:string [&lt;field: DataMember(Name = "url")&gt;] url:string [&lt;field: DataMember(Name = "attributes")&gt;] attributes:attributes [&lt;field: DataMember(Name = "country_code")&gt;] country_code:string [&lt;field: DataMember(Name = "full_name")&gt;] full_name:string [&lt;field: DataMember(Name = "name")&gt;] name:string [&lt;field: DataMember(Name = "id")&gt;] id:string [&lt;field: DataMember(Name = "bounding_box")&gt;] bounding_box:boundingbox [&lt;field: DataMember(Name = "pldace_type")&gt;] place_type:string } with member this.toCsv() = let temp = string this if temp ="" then makeString 10 else this.country+","+this.url+","+this.attributes.ToCsv()+","+this.country_code+","+this.full_name+","+this.name+","+this.bounding_box.ToCsv()+","+this.place_type [&lt;DataContract&gt;] type geo = { [&lt;field: DataMember(Name = "type")&gt;] t:string [&lt;field: DataMember(Name = "coordinates")&gt;] coordinates:float array } [&lt;DataContract&gt;] type url = { [&lt;field: DataMember(Name = "url")&gt;] url:string [&lt;field: DataMember(Name = "indices")&gt;] indices:int array } [&lt;DataContract&gt;] type hashtag = { [&lt;field: DataMember(Name = "text")&gt;] text:string [&lt;field: DataMember(Name = "indices")&gt;] indices:int array } [&lt;DataContract&gt;] type user_mention = { [&lt;field: DataMember(Name = "indices")&gt;] indices:int array [&lt;field: DataMember(Name = "screen_name")&gt;] screen_name:string [&lt;field: DataMember(Name = "name")&gt;] name:string [&lt;field: DataMember(Name = "id")&gt;] id:string } [&lt;DataContract&gt;] type entities = { [&lt;field: DataMember(Name = "urls")&gt;] urls:url array [&lt;field: DataMember(Name = "hashtags")&gt;] hashtags:hashtag array [&lt;field: DataMember(Name = "user_mentions")&gt;] user_mentions:user_mention array } [&lt;DataContract&gt;] type user = { [&lt;field: DataMember(Name = "time_zone")&gt;] time_zone:string [&lt;field: DataMember(Name = "profile_sidebar_border_color")&gt;] profile_sidebar_border_color:string [&lt;field: DataMember(Name = "screen_name")&gt;] screen_name:string [&lt;field: DataMember(Name = "notifications")&gt;] notifications:string [&lt;field: DataMember(Name = "listed_count")&gt;] listed_count:string [&lt;field: DataMember(Name = "profile_background_image_url")&gt;] profile_background_image_url:string [&lt;field: DataMember(Name = "location")&gt;] location:string [&lt;field: DataMember(Name = "statuses_count")&gt;] statuses_count:string [&lt;field: DataMember(Name = "profile_background_color")&gt;] profile_background_color:string [&lt;field: DataMember(Name = "description")&gt;] description:string [&lt;field: DataMember(Name = "show_all_inline_media")&gt;] show_all_inline_media:string [&lt;field: DataMember(Name = "profile_background_tile")&gt;] profile_background_tile:string [&lt;field: DataMember(Name = "contributors_enabled")&gt;] contributors_enabled:string [&lt;field: DataMember(Name = "geo_enabled")&gt;] geo_enabled:string [&lt;field: DataMember(Name = "created_at")&gt;] created_at:string [&lt;field: DataMember(Name = "profile_text_color")&gt;] profile_text_color:string [&lt;field: DataMember(Name = "followers_count")&gt;] followers_count:string [&lt;field: DataMember(Name = "profile_use_background_image")&gt;] profile_use_background_image:string [&lt;field: DataMember(Name = "url")&gt;] url:string [&lt;field: DataMember(Name = "friends_count")&gt;] friends_count:string [&lt;field: DataMember(Name = "profile_link_color")&gt;] profile_link_color:string [&lt;field: DataMember(Name = "protected")&gt;] Protected:string [&lt;field: DataMember(Name = "lang")&gt;] lang:string [&lt;field: DataMember(Name = "verified")&gt;] verified:string [&lt;field: DataMember(Name = "name")&gt;] name:string [&lt;field: DataMember(Name = "follow_request_sent")&gt;] follow_request_sent:string [&lt;field: DataMember(Name = "following")&gt;] following:string [&lt;field: DataMember(Name = "favourites_count")&gt;] favourites_count:string [&lt;field: DataMember(Name = "profile_sidebar_fill_color")&gt;] profile_sidebar_fill_color:string [&lt;field: DataMember(Name = "profile_image_url")&gt;] profile_image_url:string [&lt;field: DataMember(Name = "id")&gt;] id:string [&lt;field: DataMember(Name = "utc_offset")&gt;] utc_offset:string } [&lt;DataContract&gt;] type tweet = { [&lt;field: DataMember(Name = "place")&gt;] place:place [&lt;field: DataMember(Name = "geo")&gt;] geo:geo [&lt;field: DataMember(Name = "text")&gt;] text:string [&lt;field: DataMember(Name = "coordinates")&gt;] coordinates:geo [&lt;field: DataMember(Name = "retweet_count")&gt;] retweet_count:string [&lt;field: DataMember(Name = "favorited")&gt;] favorited:string [&lt;field: DataMember(Name = "source")&gt;] source:string [&lt;field: DataMember(Name = "contributors")&gt;] contributors:int array [&lt;field: DataMember(Name = "created_at")&gt;] created_at:string [&lt;field: DataMember(Name = "in_reply_to_status_id")&gt;] in_reply_to_status_id:string [&lt;field: DataMember(Name = "in_reply_to_screen_name")&gt;] in_reply_to_screen_name:string [&lt;field: DataMember(Name = "user")&gt;] user:user [&lt;field: DataMember(Name = "retweeted")&gt;] retweeted:string [&lt;field: DataMember(Name = "in_reply_to_user_id")&gt;] in_reply_to_user_id:string [&lt;field: DataMember(Name = "truncated")&gt;] truncated:string [&lt;field: DataMember(Name = "id")&gt;] tweet_id:string [&lt;field: DataMember(Name = "entities")&gt;] entities:entities } [&lt;DataContract&gt;] type fullTweet = { [&lt;field: DataMember(Name = "place")&gt;] place:place [&lt;field: DataMember(Name = "geo")&gt;] geo:geo [&lt;field: DataMember(Name = "text")&gt;] text:string [&lt;field: DataMember(Name = "coordinates")&gt;] coordinates:geo [&lt;field: DataMember(Name = "retweet_count")&gt;] retweet_count:string [&lt;field: DataMember(Name = "favorited")&gt;] favorited:string [&lt;field: DataMember(Name = "source")&gt;] source:string [&lt;field: DataMember(Name = "contributors")&gt;] contributors:int array [&lt;field: DataMember(Name = "created_at")&gt;] created_at:string [&lt;field: DataMember(Name = "in_reply_to_status_id")&gt;] in_reply_to_status_id:string [&lt;field: DataMember(Name="retweeted_status")&gt;] retweeted_status:tweet [&lt;field: DataMember(Name = "in_reply_to_screen_name")&gt;] in_reply_to_screen_name:string [&lt;field: DataMember(Name = "user")&gt;] user:user [&lt;field: DataMember(Name = "retweeted")&gt;] retweeted:string [&lt;field: DataMember(Name = "in_reply_to_user_id")&gt;] in_reply_to_user_id:string [&lt;field: DataMember(Name = "truncated")&gt;] truncated:string [&lt;field: DataMember(Name = "id")&gt;] tweet_id:string [&lt;field: DataMember(Name = "entities")&gt;] entities:entities } let decodeFullTweet (s:string) = let json = new DataContractJsonSerializer(typeof&lt;fullTweet&gt;) let byteArray = Encoding.UTF8.GetBytes(s) let stream = new MemoryStream(byteArray) json.ReadObject(stream) :?&gt;fullTweet let decodeUser (s:string) = let json = new DataContractJsonSerializer(typeof&lt;user&gt;) let byteArray = Encoding.UTF8.GetBytes(s) let stream = new MemoryStream(byteArray) json.ReadObject(stream) :?&gt;user </code></pre> <p>The next step I need to get done is to write each fullTweet record to a csv file, the issue is that sometimes some of the fields of the record type are null. For example when decoding the json string (the below is just a small part of the string ):</p> <p>rel=\\"nofollow\\">TweetDeck\",\"favorited\":false,\"in_reply_to_status_id\":null,\"entities\":{\"hashtags\":[{\"text\":\"frac1\",\"indices\":[117,123]}],\"user_mentions\":[],\"urls\":[]},\"place\":null,\"coordinates\":null,\"geo\":null,\"in_reply_to_user_id\":null,\"</p> <p>The fields "place" and "in_reply_to_user_id" will be null.</p> <p>How can I handle this so that when a field is null (for example places), the csv line reads "null" ( for each field of the record type place) ? Or if you have a better way to handle this ? </p> <p>Thanks a lot !!!</p> <p>(Hope some will enjoy all the record types :) )</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload