Using social media to identify events

(1)

Using Social Media to Identify Events

Xueliang Liu

EURECOM Sophia Antipolis, France xueliang.liu@eurecom.fr

Raphaël Troncy

EURECOM Sophia Antipolis, France raphael.troncy@eurecom.fr

Benoit Huet

EURECOM Sophia Antipolis, France benoit.huet@eurecom.fr

ABSTRACT

We present a method to automatically detect and identify events from social media sharing web sites. Our approach is based on the observation that many photos and videos are taken and shared when events occur. We select 9 venues across the globe that demonstrate a significant activity according to the EventMedia dataset and we thoroughly evaluate our approach against an official ground truth obtained directly by scraping the event venues’ web sites. The results show our ability to not only detect events with high accuracy but also mine and identify events that have not been published in popular event directories such as Last.fm, Eventful or Upcoming. In addition to the textual identification of events, we show how we can build visual summaries of past events providing viewers with a more compelling feeling of the event’s atmosphere.

1. INTRODUCTION

Today’s mobile devices and network infrastructures enable users to easily capture and distribute rich multimedia content wherever they are located. Hence, it is becoming common for people to capture and share images or videos of their every day life using their mobile phones or digital camera. A plethora of niche mobile applications such as instagr.am¹ or color² are connected to large social media applications such as Flickr, YouTube and Facebook and contribute to the exponential growth of social media data available online. How to leverage the explosion of this vast amount of data to benefit web users at large is, however, still an open and challenging problem.

In this paper, we address the problem of structuring social media activity into events and in particular how to automatically detect those events and their properties (location, time and participation). Event directories such as Last.fm, Eventful and Upcoming publish information about scheduled events. Users benefit from this information for tagging

1http://instagr.am/

2http://color.com/

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

WSM’11,November 30, 2011, Scottsdale, Arizona, USA.

and structuring multimedia material in order to ease their future retrieval. While such services are becoming increas- ingly popular, they are often incomplete, sometimes inac- curate in terms of the information they provide and always working as silos that lock information into the sites. On the other hand, they largely overlap in terms of coverage and generally fail to give an indication of users’ experience of an event’s atmosphere, while this feature is considered as of pri- mary importance to support users in deciding to attend or not an upcoming event. In contrast, one can find many photos shared online for specific events. This activity may be analyzed in order to detect and identify events occurrence and provide attractive summary. The problem we tackle is therefore how can we make use of metadata attached to media and events (tags, description, geographic location) to create the missing link between an event description and its illustrating media documents.

The paper is structured as follows. In section ??, we briefly describe the dataset used as well as the LODE and Media Resource ontologies for representing descriptions of events and media scraped from social media web sites. In section??, we detail our approach for detecting and identifying events based on user activity in social media and we show how we find more media related to detected events in order to create their visual summaries. We present some related work in section??. Finally, we give our conclusions and outline future work in section??.

2. DATA ACQUISITION

Several web sites contain information about scheduled events, of which some may display media captured at these events.

Using the public API offered by the web sites, several datasets containing event and media descriptions have been recently published [?,?]. In this section, we briefly present the Event- Media³ dataset (section ??) and we describe how we have chosen 9 venues for validating our approach (section??).

2.1 EventMedia = Flickr meets Event Direc- tories

We use part of the EventMedia dataset which is composed of more than 1,7 million photos explicitly associated to more than 110,000 events [?]. This dataset has been built from the overlap in metadata between four popular web sites, namely Flickr as a hosting web site for photos and Last.fm, Eventful and Upcoming as a documentation of past and upcoming events. Explicit relationships between scheduled events and

3http://eventmedia.cwi.nl/

(2)

photos are looked up using special machine tags such as lastfm:event=XXXorupcoming:event=XXX.

Descriptions of events are scraped from those three event directories using their API and represented according to the LODE ontology⁴, which provides a minimal model that en- capsulates the most useful properties for describing events [?].

The goal of this ontology is to enable interoperable modeling of the “factual” aspects of events, where these can be charac- terized in terms of thefour Ws: Whathappened,Wheredid it happen,Whendid it happen, andWhowas involved. Fur- thermore, the W3C Ontology for Media Resource⁵ is used to describe the media resources. This model provides properties for describing media properties such as the duration of a video, its target audience, copyright, genre, rating or the various renditions of a photo. This ontology contains a formal set of axioms defining mapping between different metadata formats for multimedia.

Figure 1: The Radiohead Haiti Relief Concert described with LODE (top) and illustrated with media described by the Media Ontology (bottom)

Figure??depicts the metadata attached to the event identified by1380633on Last.fm according to the LODE ontology. More precisely, it indicates that an event categorized as aConcerthas been given on the24th of January 2010 at 20:00 PMin theHenry Fonda Theaterfeaturing theRadio- headrock band. The link between the media and the event is realized through thelode:illustrateproperty, while more information about thesioc:UserAccountcan be attached to his URI. Hence, we see that the video hosted on YouTube has forma:creator the useraghorrorag.

2.2 Selecting Popular Venues

The EventMedia dataset has knowledge of more than 13,000 different venues for which at least one description of event explicitly associated to at least one photo is available. From this very large dataset, we selected 9 venues that proved to have a significant activity the last three years. Table ??

shows the number of events, photos and distinct users for those venues during this period according to the EventMedia dataset⁶. The ranking value corresponds to the popularity of the venue in the entire dataset when the sorting criteria is the number of distinct users that have uploaded at least one photo on Flickr taken during an event hosted at those venues.

These 9 venues are diverse in terms of the type of event they generally host ranging from music concerts and festi-

4http://linkedevents.org/ontology/

5http://www.w3.org/TR/mediaont-10/

6SPARQL endpoint: http://semantics.eurecom.fr/sparql.

Table 1: Number of events, photos and distinct users for 9 venues in the EventMedia dataset

Venue NbEvents NbPhotos NbUsers rank

Melkweg 352 6912 266 1

Koko 151 3546 155 5

HMV Forum 106 2650 130 8

111 Minna

Gallery 24 1369 105 14

Hammersmith

Apollo 79 2124 96 20

Circolo

Magnolia 79 2190 76 40

Circolo

degli Artisti 784 2590 86 43

Rotown 204 3623 49 48

Ancienne

Belgique 212 7831 56 83

vals, to circus or exhibitions. They are mainly situated in Europe (UK, The Netherlands, Italy, Belgium) except the venue111 Minna Gallerywhich is located in San Francisco (USA). More importantly, for all these venues, it is possible to get a perfect ground truth of the events they host since they all publish an up-to-date program on their web site.

We have developed different scrapers of these web sites in order to get accurate ground truth of past events⁷.

3. DETECTING EVENTS BASED ON SOCIAL MEDIA ACTIVITY

More and more media shared on the web are provided with latitude and longitude coordinates, either through user input or directly by the capture device itself. We also as- sume that when an event takes place, there will be enough persons taking picture or videos and later sharing them online. Our approach consists in measuring photo peaks at known venues in order to detect an event occurrence. It is therefore necessary to model the venue location.

3.1 Venue Bounding-Box Location Estimation

The Flickr API allows users to query photos based on geographical location. However, getting the geographical coverage or outer box of a place is not trivial. We address this issue by combining information from event and media directories. We retrieve events (EventID =E) that took place at a given venue (VenueID =V) using the Last.fm API. Then, we use the machine tags “lastfm:event=E” to search for geo- tagged media on Flickr. For each venue, a bounding box is finally computed using the GPS coordinates of the retrieved photos (algorithm??). The final bounding box is estimated as the minimized rectangle of the GPS coordinates after re- moving the outliers defined as the photos which are located further than twice the variance of the set in either direction (longitude or latitude).

Figure??shows the result of the bounding box estimation for the venueMelkweg (Amsterdam, NL). The blue marker is the GPS location of Melkweg according to Last.fm, while the red markers are the places where photos labeled with machine tags from past events have been shared on Flickr.

The red rectangle corresponds to the learnt bounding box for the venue Melkweg. We see that some photos taken too far away from the venue have been appropriately discarded.

7Scrapers available athttp://scraperwiki.com/profiles/Hou/.

(3)

Algorithm 1Estimate the bounding box for a venue 1: INPUT:V enueN ame

2: OUTPUT:BoundingBox 3: P hotoSet= [ ]

4: EventSet=GetPastEvent(V enueN ame) 5: foreacheventidin EventSetdo

6: photos= GetFlickrPhotos(eventid,hasGeo=T rue) 7: P hotoSet.append(photos)

8: end for

9: GeoSet= GetGeoInfo(P hotoSet) 10: GeoSet.filter()

11: return MinRect(GeoSet)

Figure 2: Bounding Box for the venue Melkweg in Amsterdam (NL)

We define the bounding boxes for the 9 venues selected in Table ??and we crawl all photos taken at those locations during a one month period from May 1st 2010 to May 31st 2010. A collection of 4604 geo-tagged photos has been obtained. The number of geo-tagged photos hosted on Flickr represents only a tiny fraction of all photos shared online. In order to obtain more relevant photos, we also use the venue name as a query term for retrieving photos during the same period resulting in another 4589 photos (Table??). Surpris- ingly, we notice that there are very few photos common to both queries. This shows that users seldom label their photos with a location name when geo-coordinates are available.

Table 2: Number of photos taken in the 9 selected venues during May 2010

ID Name Geotagged Photos

Venue Name

tagged Photos Duplicate Total

1 Koko 372 2040 3 2409

2 Rotown 90 273 1 362

3 Melkweg 363 700 8 1055

4 HMV Forum 184 412 0 596

5 111 Minna

Gallery 937 3 0 940

6 Ancienne

Belgique 2206 288 2 2492

7 Circolo

degli Artisti 70 553 1 622

8 Circolo

Magnolia 95 236 0 331

9 Hammersmith

Apollo 287 84 0 371

3.2 Analyzing the Flickr Activity around Venues

We are interested in detecting events by monitoring the social media sharing activity at specific locations. The 9178 photos gathered by crawling Flickr during May 2010 using either the geographical bounding boxes or the venues name

Figure 3: Event detection during May 2010 in the Melkweg (Amsterdam, NL)

will be the basis for the event detection experiments. Our objective is to identify the date and title of an event given its venue or location. Figure??depicts the daily number of photos taken at the Melkweg in May 2010 that have been shared on Flickr. One can see that the number of pictures taken varies temporally over the month. Our approach consist in carefully selecting the dates with peaks and consider those candidate date as events. More formally, let us consider the time series{di, i∈[1, T]}that represents the temporal evolution of the number of photos taken at a given venuev. The eventestarting at timetis detected when the number of photos shared is greater than a given threshold.

et= arg_i(ti> T HRESHOLD) (1) In this experiment, we compare three characteristics. The first is simply the number of online photos available for a given location and datedi=||pi||. The second attempts to incorporate the social dimension of the event by accounting for the number of different photo ownersui. The third characteristic combines the previous ones by pondering the number of photos with the number of different person that shared those photos di=||pi∗ui||. The choice of the threshold is an important factor of the events detection performance our approach can achieve. Venues hosting events have very different size and popularity. It is therefore unlikely a common threshold would provide a good and generic performance.

Here, we experiment with the average and median value of di as threshold.

3.3 Detection Results and Evaluation

We run this event detection method on the nine selected venues. Figure??depicts results for the first venue (Melk- weg): the main blue curve shows the product of the number of photos and the number of photo owners per day di, the green curve corresponds to the mean and the red one is the median of the blue curve. The stars indicate the date for which an event is detected (based on median thresholding).

The ground truth is indicated using circles over the median.

One can see that many events would be missed using the mean threshold while the median is able to capture many more events while keeping the false detection rate low.

In May 2010, a total of 242 events are reported from the official schedules of the nine selected venues, but only 150 of

(4)

them have been announced on Last.fm (and none on Event- ful and Upcoming). We evaluate the performance with re- spect to both this absolute ground truth from the official venues and from what event directories are aware of. Once an event start time is identified, we use the tags of the corresponding photos to deduce the topic of the event. All words from tags and titles of photos taken on that day are parsed and sorted by their occurrence. The top 15 keywords are kept to infer the topic of individual events. Detected events are manually matched with ground truth events based on date and title. Matching events are those sharing a common starting date and for which at least one non stop-word can be found in both the top 15 keywords list of detected events and the title of the ground truth events. Of the 150 Last.fm events, 136 can be matched to the ground truth data, using our starting date and title matching rule. When analysing our detection result, we identify 17 events that are not present in Last.fm but are present in the ground truth. In other words, our approach can automatically extend the event coverage of Last.fm with 10% new events.

Table??shows the positive detection on the Melkweg (start time and tags). For each event, the ground truth from the official venue web site and from Last.fm are compared with the most relevant tags we detect.

Figure??indicates to favor the use of the median over the mean as threshold condition for detecting events. We have identified three characteristics upon which event detection could be performed: number of image, number of owners and number of image times number of owners. Table ??

gives the results for these 6 conditions. When the median is used for thresholding, more positive events are detected compared with the ones using the mean. Unfortunately, the number of false detection also increases. Of the three upload characteristics, the combined “Image*Owner” achieves the best overall performance with an F1-Measure of 0.289 with a median based threshold. Based on these results, we decide to use the median thresholding approach on the “Im- age*Owner” upload characteristic for event detection.

Table 4: Event detection on different conditions

Source Threshold True Predict False Predict F1 Image

mean 43 21 0.211

median 64 51 0.279

Owner

mean 56 56 0.246

median 58 62 0.251

Image*Owner

mean 34 18 0.172

median 67 53 0.289

The overall detection results on the 9 venues are shown in Table??. Since the detection of events is based on social media activity, any events for which no or too few photos have been shared on Flickr cannot be detected. The recall values reported in this table should therefore be interpreted with caution. As the current sharing trends continues to evolve though many platforms, we expect the number of

“detectable” events using our proposed approach to grow accordingly. We also provide the number of events known by Last.fm for those venues during this period and we observe that our approach is able to correctly detect events unknown for this event directory (indicated with *).

3.4 Illustrating Events with Media

Event directories fail to provide viewers a good feeling about the atmosphere of events [?]. Showing photos and

Table 5: Event detection results for the 9 selected venues

VID GT

Our Method

LastFM Detect Matched P R

1 69 15 (*)12 0.800 0.174 44

2 20 15 (*)8 0.533 0.400 0

3 14 12 9 0.750 0.643 14

4 23 15 (*)2 0.133 0.087 0

5 38 15 (*)9 0.600 0.237 28

6 16 15 8 0.533 0.500 13

7 22 15 (*)8 0.533 0.364 12

8 25 3 1 0.333 0.040 11

9 15 15 10 0.667 0.667 14

total 242 120 28 0.558 0.277 136

videos related to past events is a natural way to provide more insight to viewers. We follow the approach proposed in [?] for mining and visually pruning media related to those detected events. LetF be the set of photos shared on Flickr around the date of a predicted eventi(1 day before and 2 days after) containing keywords corresponding to the event title. LetE be the set of photos explicitly linked with the eventi(using the event machine tag) from the EventMedia dataset. We select the images from Fj whose visual similarity match closely one of the images in E as additional representative media for an event. Visual similarity between images is computed as follows:

L1(Fj, Ei) =X

k

|Fj(k)−Ei(k)| (2) where Fj(k) and Ei(k) are normalized concatenating low level feature vector of the images (225D color moments in Lab space, 64D Gabor texture, and 73D Edge histogram).

Fjis added to the set of media illustrating the event when L1(Fj, Ei)< T HDi

whereT HDiis the threshold which is also learned from the E data.

T HD_i= min

{j}\i

X

k

|EventM ediaj(k)−EventM edia_i(k)| (3)

In addition, if the owner ofFjhas shared more photos during this period, they are automatically added as illustrative media for the event.

For each of the 67 events detected, we query Flickr for photos containing event specific wordsweand uploaded within a four day window around the event (1 day earlier and 2 days after). This results in a potential enrichment for 23 events (see Table ??). This table reports four different information about each individual event. The first part, “Detection Dataset”, corresponds to the data used for detecting events and from which a subset of relevant photos will be used for creating the visual model later used for pruning. The second part, “Enriching Source”, reports the number of new retrieved images which will be considered for enriching the event illustration set. The third part shows the visual pruning results. Finally, the last part gives the total number of photos that illustrate the event after enrichment and visual pruning.

Let us take the first event as an example. For this event, there are 101 photos taken on that day at this location, and 66 of them are relevant to the event (based on textual keywords match). 32 additional photos are obtained on Flickr through the enrichment process. A manual visual

(5)

Table 3: Event Detection Results on Melkweg

Venue

Detection Results Ground Truth (Official) LastFM

Date Tags Date Title LastFM Title

melkweg 03/05/2010 parkwaydrive

drive parkway 03/05/2010 Parkway Drive / Despised Icon /

Winds Of Plague / The Warriors / 50 Lions 1336473 Parkway Drive melkweg 02/05/2010 flight conchords

flightoftheconchords 02/05/2010 Flight Of The Conchords - UITVERKOCHT 1439320 Flight of the Conchords melkweg 04/05/2010 flightoftheconchords 04/05/2010 Flight Of The Conchords - UITVERKOCHT 1439407 Flight of

the Conchords melkweg 05/05/2010 mayerhawtorne

mayer hawthorne 05/05/2010 Mayer Hawthorne & The County 1416229 Mayer Hawthorne The County

melkweg 11/05/2010 bonobo 11/05/2010 Bonobo - UITVERKOCHT 1398102 Bonobo

melkweg 14/05/2010 paulweller paul 14/05/2010 Paul Weller - UITVERKOCHT 1406677 Paul Weller melkweg 18/05/2010 brokensocialscene 18/05/2010 Broken Social Scene - UITVERKOCHT 1334429 Broken Social Scene melkweg 19/05/2010 mikestern

richardbona 19/05/2010 Mike Stern band with special guest Richard Bona featuring Dave Weckl & Bob Malach melkweg 25/05/2010 beattimemelkweg 24/05/2010 Beattime - The Kika Edition melkweg 26/05/2010 beattime 24/05/2010 Beattime - The Kika Edition melkweg 28/05/2010 offcentre 28/05/2010 Off Centre - day 3 - night met Kode 9 / Falty DL / Gold Panda / Kelpe

melkweg 30/05/2010 joannanewsom 30/05/2010 Joanna Newsom 1425481 Joanna Newsom

check identified 31 of those images to be relevant to the associated event. After automatic visual pruning 20 photos are left, and all of them are relevant photos, indicating that the pruning process has discarded the non relevant photo from the enrichment set. There is a final total of 86 (66+20) photos obtained to illustrate this events. A visual representation of this event is depicted in Figure??.

In summary, for the 23 events, there are 1678 photos used for event detection, out of which 1138 are thought to be sufficiently relevant based on tag similarity to be used to model the events visually. The enriching Flickr search retrieved 632 new photos. 464 of those photos are relevant to event according to manual visual check. The automated visual pruning algorithm identifies 584 relevant photos, from which 417 are judged relevant through human inspection.

The proposed approach is therefore able to populate illustrative sets of media representing particular events with a precision of 0.714 and recall rate 0.898. In addition, the EventIDs featuring a (*) are events for which no entry can be found in Last.fm but which have been detected by our algorithm.

We generate visual clusters for each events using the method proposed in [?] (Figure??). During the enrichment phase, we expect to bring more diverse photos into the collection.

For example, the Figure ?? depicts the event 2, which is held in Hammersmith Apollowith the title iggy stooges.

Figure ??(A) is generated from the relevant photos from the detection set that corresponds to photos that are either geo-tagged or tagged with a venue name. Figure??(B) shows the collection of images resulting only from our enriching and visual pruning method. We can clearly see the increased visual diversity of the scenes between the two sets.

The final set of images illustrating theiggy stoogesevent will be composed of both sets.

4. RELATED WORK

While the earlier works aiming at event discovery were based solely on text analysis and essentially focused on news documents [?,?], event detection in social media enables to take into account rich context. For example, [?] proposed a framework to fuse tag relevance with location information and visual cues, and aimed at mining patterns from them.

In [?], Caoet al. tried to leverage the contextual information of photos in social media to benefit the annotation process.

Figure 4: Visual cluster for the event 2, which was held on 03/05/2010, in the venue Hammersmith Apollo with the title iggy stooges

First, event relation are mined from spatial and temporal metadata of photos, and then, the hierarchical structure is used to improve the simple annotation. In [?], the authors proposed a clustering framework to group images based on geographical coordinates and visual feature to depict different view of a location.

In [?], a framework to detect landmark and events to improve the user browsing and retrieval experience is proposed. The work presented in [?] attempts to identify public event using both spatial-temporal context and photo content. In [?], the authors exploit the social information from photos in Flickr in form of tags, titles and photo descriptions, for categorizing pictures into different event cat- egories. In [?], the author proposed a framework to detect events based on the assumption that documents describing the same event will contain similar sets of keywords. In [?], Becker et al. focused on identifying real-world events and their associated user-contributed social media documents automatically to enable event-based browsing and search- ing in state-of-the-art search engines. Our work differs from these systems in that we combine rich semantic information and image analysis for detecting events in social media.

Moreover, our event detection approach is not based on a user query but rather on monitoring media sharing behav- ior of users at particular location.

In some aspects, the work of Gaoet al.[?] shows interest- ing complementarities with our work. The idea of employing Twitter data to search for additional information about an event could be used in our work to extend the set of relevant words describing events. As a result, the enrichment process would retrieve an even more diverse set of candidate images.

(6)

Table 6: Illustrating events with photos

EventID VenueID

Prediction DataSet Enriching Source Pruning Results

Final Photo Nb Photos Related Photos Related Photos Precision Recall

(*)1 1 101 66 32 31 20 0.645 1 97

2 9 44 43 48 48 48 1 1 91

3 1 116 97 5 5 3 0.6 1 102

(*)4 2 161 107 21 20 20 1 1 127

5 2 61 17 1 0 1 NULL 0 17

6 6 7 1 1 1 1 1 1 2

7 1 238 201 4 0 2 NULL 0 201

(*)8 2 95 44 9 9 9 1 1 53

9 9 17 17 24 24 24 1 1 41

10 6 4 3 5 5 5 1 1 8

11 3 22 20 4 4 4 1 1 24

12 3 3 2 5 5 4 0.8 1 7

13 9 173 161 11 11 11 1 1 172

14 5 70 37 5 5 4 0.8 1 42

15 5 66 11 249 88 220 0.636 0.255 99

16 9 9 2 1 1 1 1 1 3

(*)17 2 135 53 14 14 14 1 1 67

18 6 43 41 2 2 2 1 1 43

(*)19 2 95 78 4 4 4 1 1 82

20 3 50 29 67 67 67 1 1 96

(*)21 5 53 34 10 10 10 1 1 44

22 5 52 11 54 54 54 1 1 65

23 3 63 63 56 56 56 1 1 119

5. CONCLUSION AND FUTURE WORK

We presented a novel approach for automatically detecting events and their properties (location, time and participation). The key idea of our contribution consists in temporally monitoring media shared on social web sites, registered to a specific location or referring to a specific venue. We show that using the appropriate thresholding function, it is possible to detect events in an efficient way. The result of our work can therefore be used to automatically structure online media. It was shown that the approach identified a number of events uncovered by the most popular event directories (Last.fm, Eventful and Upcoming). In addition, we have presented a process for automatically extending the number of photos illustrating an event. It consist in a two step process where first the image collection is extended through keyword search using the Flickr API before being pruned visually thanks to automatically created event visual mod- els. The resulting photo montages better capture the atmosphere of the event. The EventMedia dataset can be further enhanced with more sources and more linkage. In particular, user participation at events using public Foursquare/Gowala check-in and live tweets can be added to enrich the overall knowledge one has about a past event. Our long term goal is to provide support for exploring and selecting events and associated media, and for discovering meaningful, surpris- ing or entertaining connections between events, media and participants.