Searching Daily Kos
Quick Search Tips
Search: enter words you want to find in the target collection. By default only targets containing all the words will be returned. Capitalization is ignored. If you want to find several words in a precise order (a phrase) put double quotes around them as in "this is a phrase". Single quotes are not permitted, and are removed. Although "AND" is assumed by default, "OR" and "NOT" are supported, as follows.
Multiple query words are by default understood to be joined by "AND". Certain characters have to be "escaped" by placing a back slash in front of them; to find "/", use "\/", to find "=", use "\=". Complex queries can be constructed using parentheses and restricted queries. Restricted queries: limiting the parts of the document to be searched. The available options are described here. For both stories and comments (documents):
For stories only:
For comments only:
Find: In order to choose what kind of documents to find, use the pull down menu to select one of the following. Each option is identified as either a database search, meaning that the primary database is searched using Scoop directly, or index search, meaning that a specially constructed index of words contained in documents is searched using swish-e. There are three different indices, one for diaries (aka user diaries), one for stories (aka front page stories), and one for comments. User diaries that are elevated to the front page are found in both story and diary indices.
From: and To: pull down menus restrict the time interval of the search. The default interval is from one day ago to most recent. Most recent may be as much as 5 minutes out of date. Sort by: The order of presentation of results. Not all choices are relevant to all searches. Story and Diary Search sort choices.
Comment search sort choices shared with stories and diaries are "Relevance", "Title", and "Time", which are as described above. Those specific to comment are "Cid", "Nrec", and "Ntroll".
More example queries
For comments only Everything past the "#" is a comment, not part of the query!
Searching Daily Kos with election tags (etags)You can now search using election tags as follows. etag=CT-SEN Etags with embedded spaces should be quoted. etag="CA-LT. GOV" You can only use one etag per search. Good election tags have a two letter postal state abbreviation hyphen seperated from either a house district number, or a statewide office abbreviation (SEN, GOV, Lt.GOV, SOS, etc). There is one exception, etag="2006 ELECTION" also works. Good etags are ones we already know about by finding them in the tag list of an existing diary or by appearing in one of the election roundup diaries. The etag is used to build a more complicated query that looks for the tag in tags, title, or body text (stories/diaries) or in subject and comment text (comments). Certain meta dairies are excluded, including open threads, jotter's daily lists, and sidinny's election roundup, and the top comments diaries.
Software used.The search facility currently installed is based on the swish-e search engine: it is fast (compiled C code), open source, and flexible (you can feed it documents out of a database using your own custom preproccessor). It is intended for document collections numbering a million or less, which will do for Daily Kos with under half a million documents (excluding comments). In addition, swish-e offers many of features most desired, including search of phrases and search within configurable document areas (title, author, tags, URLs). What is indexed. All Daily Kos documents from the beginning to within 5 minutes of the present moment are indexed. Older documents are stored in static yearly indices while more recent documents are indexed and reindexed, frequently at first, then less frequently as they age, until they roll over into a growing, static, permenant yearly index. The right mix of number of index files and frequency of update to obtain the optimal combination of ease of maintenance and speed of search may require further experimentation. At present three different series of indices are maintained, corresponding to (in order of increasing total number of documents) front page stories, user diaries, and comments. For 2005 the number of stories, diaries, and comments was 5097, 88943, and 693224, respectively, containing 56993, 371578, and 2945916 distinct words. Design Considerations. The utility of search to a community of users is delivered not simply by the results of the text search, but most critically by the ordering of the results such that highest "value" hits are displayed first. (Most search researchers know that no one looks beyond about the first handful of results; almost no one looks beyond the first page.) The key to arranging for best to come first is to understand, have access to, and use the meta data associated with the text documents searched. Meta Data. At Daily Kos, we have a number of sorts of meta data that are relevant to documents. First among these is the time of posting. What is the latest? When did they say that? Who was first? That sort of thing. So the one thing search has to get right is time, in the sense of being up to date, that is including the latest diaries, as well as in the sense of permitting searches to be organized by reference to the present time. Making it up to date. Because recommendations are only open for 24 hours after a diary is published, it is important that search indices be refreshed sufficiently often during that interval to provide reasonably up to date information on total recommendations and comments. Currently, diaries up to 6 hours old are refreshed every 5 minutes, and diaries 6 to 24 hours old are refreshed every 15 minutes. More Meta Data. Other important meta data associated with stories and diaries includes comments and recommendations. The number of recommendations offers insight, imperfect, but useful, into the degree of approval a diary has received from the Daily Kos community. The number of comments indicates the size of the discussion engendered. The two numbers together give an interesting insight into a diaries history at Daily Kos. Some user diaries are promoted to the front page, and some diarists ("front pagers") are allowed to post directly to the front page. The latter are not open for recommendation. In order to harmonize listings from searches which include both front page stories and user diaries, a plausible value for number of recommendations (83) is assigned to front page stories, but in lists, this number is indicated by "*" to avoid placing too much emphasis on this made up number. The search query. Search queries can contain individual words, all of which must be found in a text document to constitute a hit. Queries can also contain phrases, indicated by double quotes around the phrase. All words are converted to lower case while searching, so letter case is of no consequence. In addition, queries can include boolean terms, such as "and", "or", and "not" which can be used to construct more complex and perhaps more informative results. By default "and" is assumed between individual words in a query. Finally, queries or parts of queries can be restricted to only search within parts of a document or document metadata. The parts available are shown above, along with examples. Certain characters have special meaning to the search engine (open and close paren, double quote, equals sign). Only a restricted list of characters are expected to occur in a word. For the indices here at daily kos, the word characters are as follows.
All other characters (non word, non special) are silently replaced by spaces while searching. For instance searching with "Jerome a Paris", "Jerome.a.Paris", or "Jerome,a,Paris" all give the same result. This is important in certain cases, for example searching for the url http://riverbendblogspot.com is accomplished with the query site=("riverbend.blogspot.com"). The reasonable looking but incorrect query site=riverbendblog.blogspot.com doesn't work as expected because in the absence of the double quotes and parentheses, replacement of "." with " " and the default boolean AND understood between search words makes it equivalent to site=riverbendblog AND blogspot AND com, which returns as hits only documents with riverbendblog in a url, and blogspot and com in the body text. More on Search
If you are intriqued by search and would
like to know more about the basics of how it is done, this is a
good introduction.
On Search. Basic basics. by Tim Bray. Since I've emphasized the importance of meta data, I'll also point to another article from the same series, On Search. SearchMeta, with some further thoughts on things meta. While I'm on the subject of underlying technologies, let me end this section by pointing to an article by Tim Bray on Ajax, the other new addition here at Daily Kos. In it, he also gives credit to Jesse James Garrett for the Ajax acronym in the process of thoroughly describing it.
Daily Kos is an amazing place, community, phenomenon. It has been my
privilege to have the opportunity to contribute.
-- jotter |
Recommended
- The Supreme Court Justice who wrote the opinion overturning Roe v. Wade displayed an upside-down “Stop the Steal”-intentioned flag at his house, less than two weeks after the Jan. 6 insurrection, purportedly in order to protest anti-Trump signs a...138 comments 375 Recs
- Ukraine hit the Belbek Air Base in Crimea for the second night in a row. Your US tax dollars being put to good use. It’s a pity there is no video. We might have seen another appearance by Ammo Dump Jesus. I know an even better way — fly them back to...82 comments 383 Recs
- It baffles me why Democrats and the media are ignoring the most devastating act of Espionage in US history, by a long shot. The timeline of Trump’s Mar-a-Lago “mishandling of classified documents” is a series of inexplicably indifferent decisions to...122 comments 389 Recs
- The three-day weekend shows the defense on cross-examination trying to get that one juror to cause a mistrial at the hush-money event. Today’s oral testimony extended into the street. Apparently, Trump never would have paid off the porn star to shut...78 comments 205 Recs
- NFL kicker's commencement speech slams LGBTQ, feminism, contraception, choice, IVF, the media + JewsHarrison Butker is a placekicker for the Kansas City Chiefs and a miserable human being. NFL fans will probably know him. Everyone else will likely not. However, after his commencement speech at Benedictine College, he will be more famous — or infamous...616 comments 525 Recs
- There’s a supporting blogpost and transcript about this cartoon here; I’ll also post the transcript in comments. We can keep making these cartoons because of lots of supporters pledging low amounts - $1-$3 - and that's just how I like it! Please peruse...49 comments 311 Recs
- Many of you contributed to my GoFundMe drive a few months ago, and thank you all again for your support! The drive’s goal was $2,000 to allow me to buy a new book for each student in my elementary school, and once I posted about it here on DKos, that...17 comments 129 Recs
- Note: The reader should take nothing in the following diary as a criticism of any individual Catholic and their faith. The Constitution, and more importantly decency and respect, protect every American’s right to worship as they see fit — or not at...49 comments 108 Recs
- From the Guardian: “Had I been President Biden,” Romney said, “when the justice department brought out indictments, I would have immediately pardoned him. I’d have pardoned President Trump.” “Why? Well, because it makes me, President Biden, the big guy...319 comments 431 Recs
- I usually wouldn’t post a diary about polling, but this is actually quite important. Typically, I’m not the sort of person who celebrates polls that make me feel good and attacks ones that don’t, as a) I tend to look at the overall average and trending...207 comments 561 Recs
- The GOP’s favorite phrase when it comes to presidential politics appears to be, “Nobody knew at the time.” Once a president is sworn into office, regardless of how much evidence there is of crimes and irregularities committed to get him there, both the...122 comments 462 Recs
- In the chicken slaughtering industry, the term “cadaver” has a different meaning. Also called “red birds,” “cadaver” refers to chickens who drown in scalding water during the slaughter process. It’s easy to tell when this happens because when birds are...118 comments 255 Recs
Recent Diaries
- Welcome to White House Picture of the Day Showcasing a photo from the White House Photography staff, led by Adam Schultz. Feel free to join us anytime and share with us what is happening in ...2 comments 6 Recs
- The Russians are REALLY lucky Ukrainians are not like Russians. Really damn lucky. Unlike the Russians, the Ukrainians have chosen not to hit civilian facilities but have instead focused their air borne assaults on critical Russian petrochemical...4 comments 12 Recs
- As soon as I heard they had agreed to a debate, this image just popped into my head.1 comments 0 Recs
- 1 comments 1 Recs
- Happy Friday! I have a doctor’s appointment this morning, and they are going to be working on the net in our neighborhood as well, so if I don’t get online, please be kind to each other.� Above ...14 comments 8 Recs
- Ukrainian officials reported that Ukrainian forces are stabilizing the situation along the northern border in Kharkiv Oblast and that the tempo of Russian offensive operations in the area continues to decrease. NATO allies are inching closer to sending...18 comments 50 Recs
- Tonight’s selections from New Order’s fifth album, 1988’s Technique . Start to finish, an excellent album. A perfect album of 80s synth pop, rock and dance tunes. Upon its release in January ...13 comments 10 Recs
- Charter Schools are among the biggest frauds in education. After his Houston Independent District coup, TX Gov. Abbott installed Mike Miles, a former failed Charter School administrator. Texas public funds were funneled to his out-of-state Charter...3 comments 15 Recs
- Guest: Campaign finance expert Craig Holman of Public Citizen Also: Two (generally) encouraging SCOTUS rulings on CFPB funding and a second Black-majority U.S. House district in Louisiana... It's received almost zero attention or notice, but the...3 comments 4 Recs
- A look at my favorite commercial TV music show, after-the-jump …. But first: Top Comments appears nightly, as a round-up of the best comments on Daily Kos. Surely ... you come across comments daily that are perceptive, apropos and .. well, perhaps even...4 comments 18 Recs
- Yes, there are many many classic movies about cats. And dogs. And jungle critters. And we are here for all of them Animated ones like: The Lion King The Jungle ...33 comments 50 Recs
- Just a few updates for those interested:\ Gil the Wonder Cat is still very much with us. He’s eating, drinking, peeing, pooping, and grooming his beautiful soft fur, and if he’s thinner than before and a bit shaky, well, he’s 13 years old. He’s clearly...26 comments 67 Recs
- This lifelong Democrat has decided to run for office! High on the heels of my victory as a District Delegate to the DNC in Chicago this August, I’ve decided to run in my district to flip District 68 in Florida blue! I have strong Democratic values- I...10 comments 19 Recs
- 3.5 of 5 stars Fourth Wing (The Empyrean #1) by Rebecca Yarros I picked up a ...2 comments 9 Recs
- The Supreme Court Justice who wrote the opinion overturning Roe v. Wade displayed an upside-down “Stop the Steal”-intentioned flag at his house, less than two weeks after the Jan. 6 insurrection, purportedly in order to protest anti-Trump signs a...138 comments 375 Recs
- Not much going on this past week. I’ve been mostly working on my living trust, which I had prepared for me by the online legal service Trust&Will at the advice of my financial advisor. For a ...50 comments 21 Recs
- Note: The reader should take nothing in the following diary as a criticism of any individual Catholic and their faith. The Constitution, and more importantly decency and respect, protect every American’s right to worship as they see fit — or not at...49 comments 108 Recs
- Hi all, long time no see! Life keeps interfering with my writing goals, though in April I�made a little progress on that front. Slow and steady wins something eventually, right? Come on in, and let’66 comments 14 Recs
- Welcome to the Weekly Fiction WIP Thread!� This is a weekly community diary for writers here on Daily Kos to showcase microfiction, ongoing serialized stories, and pieces of larger works in progress.�21 comments 7 Recs
- Many of you contributed to my GoFundMe drive a few months ago, and thank you all again for your support! The drive’s goal was $2,000 to allow me to buy a new book for each student in my elementary school, and once I posted about it here on DKos, that...17 comments 129 Recs