Discovery in a federated world
Laurent Eschenauer
In this review of Diaspora by Mashable, I was surprised to see that
the #1 complain about the service with the lack of discovery:
"What makes Diaspora worse, however, is that there is no easy way to
find people. OK, you can search by Diaspora username and if a person
has opted, you can search first and last name in the directory, but
you can’t compare your contacts from Twitter or Facebook against the
Diaspora database."
http://mashable.com/2010/11/24/diaspora-preview/
Thinking about it, it makes perfect sense from a user point of view.
You just want to get your twitter, email or Fb contacts and find them
on the new network you are trying out. From a tech side point of view
however; I see this problem as a very hard one to solve in a
decentralized world. Doing a search on a username, or even a name
which would be public is easy (Google does the job), but how to do it
on private data like email, facebook account, etc... ?
I was just wondering if anyone had already started looking into this.
How would this be done in federated status.net for example ?
Cheers,
Laurent
Michael Chisari
Michael Chisari
The Appleseed Project
Andrew Rondeau
Bing and a ton of other me-too search engines. Building a web crawler
and search engine into a simple web server is a little impracticable,
so I'm of the opinion that the solution will be somehow involve
introducing "type" into search engines and web sites. IE, searches
will be redirected to a major search engine with some kind of flag
saying, "this search is for user profile pages."
Something that I'm considering experimenting with in ObjectCloud
(http://objectcloud.kicks-ass.net/) is to just hide the word
"76rdfi7jeny" on user profiles, and then do "search" by appending the
the word "76rdfi7jeny" to the search query and send it to Google or
Bing.
So, if you were to search for the user "Andrew Rondeau," I'd just
forward you to a Google or Bing search for "Andrew Rondeau
76rdfi7jeny"
Yes, it's a bit of a hack, but it's pretty simple to implement.
On Nov 25, 1:17 pm, Laurent Eschenauer <laurent.eschena...@gmail.com>
wrote:
Evan Prodromou
There's really no easy answer for this. We already have Google and Bing and a ton of other me-too search engines. Building a web crawler and search engine into a simple web server is a little impracticable, so I'm of the opinion that the solution will be somehow involve introducing "type" into search engines and web sites. IE, searches will be redirected to a major search engine with some kind of flag saying, "this search is for user profile pages."
http://code.google.com/apis/socialgraph/
It exposes data from real Google crawls. Great to work with.
Yahoo's SearchMonkey and BOSS might have worked for this, but since they're deprecated now, I don't know.
I haven't been able to find any details on using the Bing API for similar things.
-Evan
|
-- Evan Prodromou, CEO, StatusNet Inc. 1124 rue Marie-Anne Est #32, Montreal, QC, Canada H2J 2B7 E: ev...@status.net T: 438-380-4801 x101 W: http://evan.status.net/ |
Michael Chisari
Maxi
I think it is vital that the results show up instantly, as people don't
have patience when searching nowadays. So in my opinion, the second
approach is not sufficient as long as we can't solve this
problem(perhaps you have an idea?).
However, a simple database containing only let's say the full name,
hometown and a webfinger address of a person will not take so much
space. Even when we have as much users as Facebook some day, the data
will still fit on a disk with some hundreds of gigabytes, plus some
overhead for an index - and when this day comes, disk space will
probably be much cheaper anyway :-)
So I think we do not necessarily have to split the database between many
servers, but instead can simply synchronize it, so that each server has
a full copy of the database and can search quickly.
It is not a good idea to integrate the software fulfilling this task
into each of Appleseed, diaspora, onesocialweb and so on. The best way
is to have a pool of database servers synchronising with each other, who
provide a common querying API for all projects.
I have already started to program a software for this task; the project
page is here: https://github.com/Leberwurscht/Diaspora-User-Directory/wiki
It's currently called "diaspora user directory", but the plan is to
index webfinger pages, so it can be used for any social networking software.
I'm curious about you to publish your concept. I am very interested in
seeing this problem solved, and would like to work together with you.
Maxi
Melvin Carvalho
On Fri, 2010-11-26 at 14:55 -0800, Andrew Rondeau wrote:I hope everyone looking into this issue considers the Google Social Graph API:There's really no easy answer for this. We already have Google and Bing and a ton of other me-too search engines. Building a web crawler and search engine into a simple web server is a little impracticable, so I'm of the opinion that the solution will be somehow involve introducing "type" into search engines and web sites. IE, searches will be redirected to a major search engine with some kind of flag saying, "this search is for user profile pages."
http://code.google.com/apis/socialgraph/
It exposes data from real Google crawls. Great to work with.
Yahoo's SearchMonkey and BOSS might have worked for this, but since they're deprecated now, I don't know.
I haven't been able to find any details on using the Bing API for similar things.
SIndice also has an API for open public structured relationship data
http://sindice.com/developers/welcome
Daniel Renfer
Finding other social web users will be as easy as giving some website access to your gmail address book and getting back a list of users to follow. (assuming Google doesn't offer this service directly)
Raphael Sofaer
Laurent Eschenauer
There will still always be a need to search by name, by attributes, etc, rather than by unique identifier.
Andrew Rondeau
We still aren't seeing a protocol emerge that handles distributed
social networking without search. Considering the challenges in
federating a protocol without search; "designing by committee" some
kind of decentralized profile search engine is going to take a long
time.
Likewise, a centralized database of profiles is going to be difficult;
is everyone going to agree on who hosts it?
I'm a big fan of letting 2-3 well-designed approaches compete.
Perhaps, instead of trying to design the entire thing by committee, we
can figure out very simple ways to flag that "a profile is a profile"
and then see which search systems actually "win in the marketplace?"
For example, I see this as merely a matter of:
1: Figure out a way to flag a profile so that a web crawler knows what
to do with it. (IE, something better then my "76rdfi7jeny" trick.)
2: Figure out a way to push a profile to known search engines.
Distributions could then maintain lists of search engines that they
push to, and update the lists with each release.
In general, this allows a lot of experimentation with how a search
engine is designed, and what features it can support.
On Nov 27, 2:27 pm, Laurent Eschenauer <laurent.eschena...@gmail.com>
wrote:
>
> > There will still always be a need to search by name, by attributes, etc,
> > rather than by unique identifier.
>
> Agree. And some of these will be more 'private' (e.g. a search based on an
> email address) and thus not indexable in a regular public search engine.
> You would have to search based on hashes as not to reveal what you are
> searching for.
>
> I personally see the need of a 'search protocol' in the stack, next to the
> activities, relations, profiles, etc... Most likely based on some sort of
> distributed hash table.
>
> -Laurent
>
>
>
> On Sat, Nov 27, 2010 at 12:30 PM, Daniel Renfer <d...@kronkltd.net> wrote:
>
> >> Federated search (at the node level) won't be as important as more and
> >> more people move towards using email-like identifiers for their social web
> >> identity. We already have tons of ways of relating our contacts to an email
> >> address and as more domains begin to offer a social web presence or at the
> >> very least a way to link to a service that does host that presence tied to
> >> an "email address" it will be even better. There are already several ways to
> >> do that and as adoption increases, I'm sure you'll begin to see more
> >> services that index the capabilities of a user@domain identifier.
>
> >> Finding other social web users will be as easy as giving some website
> >> access to your gmail address book and getting back a list of users to
> >> follow. (assuming Google doesn't offer this service directly)
>
> >> On Sat, Nov 27, 2010 at 12:19 PM, Melvin Carvalho <
> >>> On 27 November 2010 02:58, Evan Prodromou <e...@status.net> wrote:
>
> >>>> On Fri, 2010-11-26 at 14:55 -0800, Andrew Rondeau wrote:
>
> >>>> There's really no easy answer for this. We already have Google and
> >>>> Bing and a ton of other me-too search engines. Building a web crawler
> >>>> and search engine into a simple web server is a little impracticable,
> >>>> so I'm of the opinion that the solution will be somehow involve
> >>>> introducing "type" into search engines and web sites. IE, searches
> >>>> will be redirected to a major search engine with some kind of flag
> >>>> saying, "this search is for user profile pages."
>
> >>>> I hope everyone looking into this issue considers the Google Social
> >>>> Graph API:
>
> >>>>http://code.google.com/apis/socialgraph/
>
> >>>> It exposes data from real Google crawls. Great to work with.
>
> >>>> Yahoo's SearchMonkey and BOSS might have worked for this, but since
> >>>> they're deprecated now, I don't know.
>
> >>>> I haven't been able to find any details on using the Bing API for
> >>>> similar things.
>
> >>> SIndice also has an API for open public structured relationship data
>
> >>>http://sindice.com/developers/welcome
>
> >>>> -Evan
>
> >>>> --
> >>>> Evan Prodromou, CEO, StatusNet Inc.
> >>>> 1124 rue Marie-Anne Est #32, Montreal, QC, Canada H2J 2B7
Mike Macgirvin
> 1: Figure out a way to flag a profile so that a web crawler knows what
> to do with it. (IE, something better then my "76rdfi7jeny" trick.)
link rel="lrdd" with an href (not a template) might be a pretty good
indicator of a federated profile - and many of us have this today. And
since this leads directly to the personal XRD, all of the information a
directory might need to render the entry is discoverable.
There's probably a good commercial opportunity for somebody to run with
a profile search - and it's also a natural fit for Google's portfolio.
That's why my own service is providing a service-wide directory only as
a temporary measure until such time as the federated social landscape
achieves critical mass and somebody seizes this opportunity.
So the real issue is how can we best inter-operate our service-wide
searches until the right candidate emerges - and acknowledging that
whatever we do doesn't have to be the perfect solution because it
probably is temporary.
One thing my service does provide is a meta tag to indicate that this
person doesn't wish to be indexed in the directory. This is something
that should also be standardised - sooner rather than later.
Laurent Eschenauer
> indicator of a federated profile - and many of us have this today. And since
> this leads directly to the personal XRD, all of the information a directory
> might need to render the entry is discoverable.
This is based on the assumption that profiles are public and
crawalable. I challenge that. Users of a social network want their
profile to be private, yet to be discoverable to users of the network
who may know some bits about them. Isn't that the whole point of
facebook ? This is also one of the challenges the Diaspora team now
has to solve.
Relying on public data and centralized search engines won't be sufficient.
Laurent
B. Kip
Brad
Mike Macgirvin
>>> good indicator of a federated profile - and many of us have this
>>> today. And since this leads directly to the personal XRD, all of
>>> the information a directory might need to render the entry is
>>> discoverable.
Laurent:
> This is based on the assumption that profiles are public and
> crawalable. I challenge that. Users of a social network want their
> profile to be private, yet to be discoverable to users of the
> network who may know some bits about them. Isn't that the whole point
> of facebook ? This is also one of the challenges the Diaspora team
> now has to solve.
>
> Relying on public data and centralized search engines won't be
> sufficient.
>
> Laurent
Yes, an assumption on my part based on our unique service architecture.
I would have to agree - it will probably not work in many other services.
Brad:
> If we are talking about discovery in the context of a federated
> social web, then how much meaning does 'discoverable to users of the
> network' have? Do you mean users of the same application? Users
> will need to be able to determine what information about them is
> available to whom. Some will want their name and social web
> identifier public so anyone can find them. Some will want more
> information than that to be public. Some will want even that
> information available only to specific people, like a subset of
> 'friends of friends'. How to give users that control, and then how
> to perform searches that relate the identity of the searcher to the
> searched- all across the federated web- are difficult, but
> *important* questions to address.
We have one profile which is public and cannot be disabled (although it
may be excluded from public listing in site and service-wide directory
services). This public profile can be as sparse as desired.
We have unlimited profiles which may be tailored to different audiences
and contain content relevant to the intended audience. Every friend may
be assigned to one of these profiles. For instance one profile might
contain your email address, and one might list you as "single" while a
different one presents you as "it's complicated". One profile might
contain your location at the street level and another only by country.
As the proposal mentioned at the head of this message is not universally
acceptable, I will withdraw it - although the lrdd auto-discovery
information is already present in many federated profiles and one cannot
prevent anybody from exploiting it.
Melvin Carvalho
Agree that centralized search is not the endgame.
But Skype got to 500 million users with a centralized public search.
Facebook is smarter in that it delegates the trust of your social network.
Eventually we should build an open federated version of both systems,
but I think the public version is going to be the easier one to start
with.
As for identifiers, easiest to go by is name, probably then email
address, but a profile page or other universal identifier like a phone
number may at some stage be used. I think the HTML5 data layer is
going to make it much easier to markup this info in profiles. We need
a smarter trust infrastructure, but I do feel problems such as this
can be incrementally solved as the infrastructure matures .
>
> Laurent
>
Jon Spriggs
The basic idea is that your social service updates at least one, but preferably three separate directory services with a path to your profile, and the data be cryptographically signed and dated (with a "Best Before End" expiry date) before updating to ensure ownership and validity.
When looking for someone, Alice asks Alice's preferred directory server for where to find Bob. Alice's directory service replies that it doesn't know where to find Bob, but here's a list of other servers where you might find Bob in one of them. The theory is that Alice probably should never need to manually say "Oh no, Bob's not on server 1, let's try server 2", but the software being used to look for Bob should perform these lookups automatically.
The speed is likely to be an issue, so Alice's system should probably try and cache all of Alice's existing contacts FoaF pages as well, so when Alice looks for Bob, it checks all of Alice's contacts *first* for whether they know Bob.
The plan with this was to have the directory services use TOR, but have at least one, if not several public facing seed lists to supply the TOR addresses of the directory services it knows about. The service should also use TOR as a back-end to exchange "I am up" data between each other, and to share known server lists and trustworthy status, given that A) it should be relatively difficult to generate TOR hidden service addresses and B) any data shared across this link should also be cryptographically signed before being shipped into the ether.
The concept is, I believe, relatively good, but I'm more than happy to be proven wrong on this. I'm imagining it to be vaguely like Facebook's "Find someone you know" system, but distributed.
When pitching something similar to this a few years back for Laconica (as was at the time), Evan said that it wasn't a very efficient system... but then again, I don't recall *exactly* what I'd pitched at the time :)
[1] The project (or rather, holding page) is at http://gitorious.net/socialservicefinder
--
Jon "The Nice Guy" Spriggs
Martin Atkins
> On Fri, 2010-11-26 at 14:55 -0800, Andrew Rondeau wrote:
>> There's really no easy answer for this. We already have Google and
>> Bing and a ton of other me-too search engines. Building a web crawler
>> and search engine into a simple web server is a little impracticable,
>> so I'm of the opinion that the solution will be somehow involve
>> introducing"type" into search engines and web sites. IE, searches
>> will be redirected to a major search engine with some kind of flag
>> saying,"this search is for user profile pages."
>
> I hope everyone looking into this issue considers the Google Social
> Graph API:
>
> http://code.google.com/apis/socialgraph/
>
> It exposes data from real Google crawls. Great to work with.
>
> Yahoo's SearchMonkey and BOSS might have worked for this, but since
> they're deprecated now, I don't know.
>
> I haven't been able to find any details on using the Bing API for
> similar things.
>
A long time ago I made a mashup of Social Graph API and Google's web
search API to make a "People Search" app:
http://martin.atkins.me.uk/peoplesearch/#david%20recordon
There are some weird bugs in it -- in this case it returned two of David
Recordon for some reason -- but it shows that you can combine
traditional web search with the social graph API to get people-oriented
search results, at least for people who make a graph of their profiles
with XFN.
(I notice that it can't find me anymore because my personal website has
been down for a bunch of time, but that's my problem.)
Andrew Rondeau
> crawalable. I challenge that. Users of a social network want their
> profile to be private, yet to be discoverable to users of the network
> who may know some bits about them. Isn't that the whole point of
> facebook ? This is also one of the challenges the Diaspora team now
> has to solve.
users have access to. ;) The biggest challenge is figuring out how to
impersonate your users when crawling private information on other
peoples' profiles.
Daniël Bos (远洋)
Did I just hear "WebID"? ;-) Using WebID would make it trivially easy to impersonate users, and provides additional encryption (SSL client certificates) on the way, so no eavesdropping!
B. Kip
Question, maybe stupid:
My understanding of 'impersonate' is to pretend you are someone else- bad thing in this context (security fail). But I get the sense this is used with a different meaning here- can you translate?
Brad
Daniël Bos (远洋)
Hi,
You are quite right that 'impersonate' usually means something bad. In this sense however it should be understood as "act on behalf of". When a user has certain friends on the internet, the node on the social network he is using had to act on his behalf when talking to other nodes. While doing so, it has to "prove" the user has authorised him to do so.
On Tue, Nov 30, 2010 at 12:53, B. Kip <bkpu...@gmail.com> wrote:
> Hi,
>
> Question, maybe stupid:
>
> My understanding of 'impersonate' is to pretend you are someone else- bad
> thing in this context (security fail). But I get the sense this is used
> with a different meaning here- can you translate?
>
> Brad
--
远洋 / Daniël Bos
email : cor...@gmail.com
phone : +31-318-711063 (Dutch) / +86-18-701330735 (Chinese)
weblog : http://blog.loadingdata.nl/
ostatus: cor...@status.loadingdata.nl
B. Kip
Melvin Carvalho
> Did I just hear "WebID"? ;-) Using WebID would make it trivially easy to
> impersonate users, and provides additional encryption (SSL client
> certificates) on the way, so no eavesdropping!
Just to note that implementations like status.net, livejournal, hi5
etc. have been issuing WebID's for a long time. If you have a FOAF
profile, you have a WebID. Indeed, livejournal's WebID implementation
(Yadis) was the 'father' of OpenID!
For operators switching to SSL (e.g. to protect against MITM /
firesheep) you would just need to add a public key to your profile to
become a WebID identity provider. This should theoretically be as
easy as adding one extra attribute to your profile's markup, once
HTML5 starts to gain some more traction.
Kingsley Idehen
Hi,
You are quite right that 'impersonate' usually means something bad. In this sense however it should be understood as "act on behalf of". When a user has certain friends on the internet, the node on the social network he is using had to act on his behalf when talking to other nodes. While doing so, it has to "prove" the user has authorised him to do so.
Links:
1. http://esw.w3.org/WebID - WebID Wiki Page
2. http://getwebid.org/spec/drafts/ED-webid-20100809/index.html -- Spec
Kingsley
On Tue, Nov 30, 2010 at 12:53, B. Kip <bkpu...@gmail.com> wrote:
> Hi,
>
> Question, maybe stupid:
>
> My understanding of 'impersonate' is to pretend you are someone else- bad
> thing in this context (security fail). But I get the sense this is used
> with a different meaning here- can you translate?
>
> Brad--
远洋 / Daniël Bosemail : cor...@gmail.com
phone : +31-318-711063 (Dutch) / +86-18-701330735 (Chinese)
weblog : http://blog.loadingdata.nl/
ostatus: cor...@status.loadingdata.nl
-- Regards, Kingsley Idehen President & CEO OpenLink Software Web: http://www.openlinksw.com Weblog: http://www.openlinksw.com/blog/~kidehen Twitter/Identi.ca: kidehen
Kingsley Idehen
How about "Agency" instead of impersonation? Basically, "Agency" is inherently verifiable via agent WebID and WebID authentication protocol combo.
Links:
1. http://esw.w3.org/WebID - WebID Wiki Page
2. http://getwebid.org/spec/drafts/ED-webid-20100809/index.html -- Spec
Kingsley