The Adblock Project

[Bethrezen]

Hi. I just tried the latest version of adblock, and something's not right. When set to collapse images, it's removing things it shouldn't. For instance, with WildersSecurity.com, I have this filter set to block the images -- so the forum loads faster, but the buttons remain intact: wilderssecurity.com/YaBBImages/*.jpg

However when I set adblock to collapse the blocked images, it also removes the forum titles. What I should see is something like this

New Members and Help
-Test Forum
-FAQs

But what I actually see is this:

-Test Forum
-FAQs

Note the forum title has vanished

This is an easily re-creatable bug. Just go to that site, set adblock to collapse blocked images and add the filter. The thing that is really annoying me is that the same filter in the v.3 didn't cause this behaviour. The page loaded, the images collapsed and that was it.

Originally, I posted here, and the response I got was basically that this is expected functionality.

However, this is causing EXTREMELY ANNOYING PROBLEMS. The issue above isn't limited to that site -- its happening everywhere I go, messing up pages' functionality. For example: buttons you need to operate a page vanish.

[edited for clarity - rue]

[Bethrezen]

[rue]

Bethrezen:
When it comes to creating filters, the user takes the full burden of properly restricting. If a filter is too general, it blocks desired content.
.
Adblock isn't doing anything wrong. Your filter just isn't specific enough. Here's one that is:
/wilderssecurity\.com\/yabbimages\/(.(?!bg))*\.jpg/

.
(tip: click the "adblock" status-element to see all blockable elements in a page.)

[Bethrezen]

hi

well if its the case that its just my filters that ain't good enough would you care to educate me coz i read that page on regular expressions and its all double dutch to me

and please keep it simple

in-fact while I'm on the subject isn't there an easer way to get the same results rarther than using regular expressions because regular expressions are beyond the capability of most novice users like my self

maybe its all very easy to people that understand all this stuff but ya need to keep in mind that not all users are wiz kid programmers and like me they probably got no idea how to use regular expressions

so is there any chance of simplifying things a lil bit to make it easer for the non technical user

[rue]

Bethrezen:
After reading this, go back to the RegExp page and "see with new eyes." Here we go. All patterns must begin and end with the forward-slash "/". This is so that javascript can distinguish them as patterns. Pattern matching works by using special characters, whose meanings you must memorize. These special characters only match positive on certain kinds of text. The two we'll cover here are the word-character match "\w" and the digit-character match "\d".
.
Example-I: I have a string containing a number: "3".
Using the digit-character /\d/, results in a positive match.
.
Example-II: I have the same string: "3".
Using the word-character /\w/, results in a negative match.
.
Example-III: Finally, I have a string containing a letter: "c".
Using the word-character /\w/, I now have a positive match.
.
Real-world Example: A website I frequent has banner-ads with their banner-size spelled out in the source:

"http://coolsite.com/notabanner183x64.gif"
"http://coolsite.com/innocentpic123x72.jpg"

Since I'd rather not block the whole site, what can I do? The answer: use a pattern. This simple RegExp will match true on their "size"-string: /\d\d\dx\d\d/ -- that is: three digits, followed by a lowercase 'x', and two more digits.

[Guest]

hi

Ok i read your explanation and i understood besicley you are telling the computer to match anything in the url that is formulated like this 183x64

but that still don't really help because it doesn't really tell me how to properly formulate filters

take for instance the one you posted for wilders even after reading i still cant figure out how or why that works and how to replicate thoughts resultes on other sights

like i asked before isn't there a way to make this any easer because this is frustratingly complicated and i'm still not getting how the hell this regular expressions thing works and i'm sure i'm not the only one

even after reading its still double dutch to me

[cardinal]

Guest/Bethrezen,

Your frustration with regexps is understandable. But, regexps are extremely powerful when it comes to pattern matching, and so the effort to learn them is well worth it, IMHO.

As rue said above, you need to memorize the special codes, or keep a bookmark to a page that has them. Or, just buy a book. The so-called "camel book" on Perl programming actually has a very good list of simple regexps for beginners, and is worth buying. The full citation is "Programming Perl", by Larry Wall et al., published by O'Reilly and Associates, Inc.

As for the specific filter rue gave that you didn't understand? Here's a brief explanation, although I'm not sure it'll do you much good.

/wilderssecurity\.com\/yabbimages\/(.(?!bg))*\.jpg/

The forward-slashes at the beginning and end are to indicate that the filter is a regexp. That's all they do.

The period (.) is a special character (a.k.a. "metacharacter" in regexp lingo), the ultimate wildcard in regexps that matches absolutely anything (even blank space). So, if you just want to match on a plain old period, you need to precede it with a backslash. This is a general rule in regexps - any metacharacters (there are like a dozen or so metacharacters, including slashes, parentheses, asterisk (*), exclamation (!), question (?), the backslash itself, obviously, and others) need to be preceded by a backslash to keep their literal, as opposed to special, meaning.

The * in regexps means something different than you might think - it matches 0 to infinite occurences of whatever immediately precedes it. So, the regexp /\d*/ would match any number of digits in a row (including no digits at all! This is why using * in regexps is somewhat tricky), since "\d" is the code for a numerical digit.

So consider the expression /.*/ This means to match on 0 to infinite occurences of any character at all. In other words, absolutely everything! I don't recommend using this as an Adblock filter

Rue's filter above simply uses a lot of "\." to implement a literal period in the web address, and "\/" to implement literal forward-slashes. The real guts of the filter is the part near the end:

(.(?!bg))*\.jpg

This uses a special construct called a "zero-width negative lookahead", which is made by an open parentheses, question mark, exclamation point, <stuff>, close parentheses. In other words, "(?! stuff )". What this does is match on anything except phrases that end in "stuff". So, in rue's filter, he uses /(.(?!bg))*\.jpg/ to match on all .jpg files, except those that have names ending in the letters "bg". This is because if you look at the wilderssecurity web site source code, all of the images that you want to preserve on the page seem to end in "bg.jpg".

So, that's it! It looks very complicated, but it's not that bad, really, once you learn to see past all the spurious backslashes

Hope that helps. Trust me, regexps can get a lot more complicated (and more simple!) than this, but, once again, are well worth learning.

[Guest]

From an Adblock-marketing standpoint, it might be better to emphasize wildcard blocking before regexps. I've pitched Adblock to my friends many times and they seem to respond more positively to "block ads with wildcards, and by the way it has regexps" than to "block ads with regexps, and by the way it has a simplified wildcard system".

Just my two cents,

James

[rue]

Bethrezen:
Alright, so now we understand how to accomplish basic matching. What about more complex stuff -- like "oneThing" OR "another"; and how can I exclude certain matches, so not everything is caught? The answer involves two, related concepts: conditionals, and sets.


-Conditionals are special characters that insert logic into your pattern.
Example-IV: I have three strings: "place", "placed", "placard"
Using the OR-conditional /place|placed/, results in positive matches on the first two, but not the third.


-Sets are simply the arrangement of one-or-more conditionals inside a parenthesis.
Example-V: I have the same strings: "place", "placed", "placard"
Using the OR-conditional, this time inside a set /plac(e|ed)/, results in positive matches on the first two, but not the third.


-Now, let's make the pattern explicitly exclude the third string, instead of spelling-out the first two.
Example-VI: again: "place", "placed", "placard"
Using the negative-lookahead /plac(?!ard)/, results in positive matches on the first two, but not the third.


Ah- so this is pretty easy. Is there any way to make whole sets conditional?- so that, maybe, nothing in them has to match? Yes- for this, we use multipliers.

-Multipliers allow items to be matched multiple-times -- with precision. Whatever directly precedes the multplier is affected. The most common multiplier is the "wildcard" (*). It matches zero or more times.
Example-VII: I have three strings: "/YaBBImages/wild.jpg", "/YaBBImages/wilders.jpg", "/YaBBImages/catbg.jpg"
Using the word-character and wildcard multiplier /wild(ers)*\.jpg/, results in positive matches on the first two, but not the third.

[Note: the period and forward-slash are special-characters in regexp. To access their "literal" meaning, we precede them with a back-slash. This is called literalizing.]


-Finally, let's bring it all together:
Example-VIII: the same strings: "/YaBBImages/wild.jpg", "/YaBBImages/wilders.jpg", "/YaBBImages/catbg.jpg"
Combining everything we know /YaBBImages\/(\w(?!bg))*\.jpg/, results in positive matches on the first two, but not the third.

[rue]

Cardinal's post is spot-on, but Bethrezen stressed the need for simplicity.
.
Between our three posts, I think everything's been covered, now.

[Melon]

Not sure about Bethrezen, but superb explainations Rue and Cardinal. I know absolutely no RegEx, but those 2 posts were very helpful.

Actually makes RegEx look somewhat less daunting, well, only slightly. Though I'll still probably be tearing my hair out once I actually try to construct my own filter

Kudos to you two.

[Melon]

Heh, told you I'd need help. Right I've modified a filter you posted on another thread Rue:

/[\W\d]banner(s|id\=)[\W\d]/

By my understanding, the first bit in the square brackets are a set and it matches any non-word character and a digit character. Followed by the main part, matching banner or banners or bannerid. Not quite sure what the \= is but I thought maybe it also covers banner= . Then the non-word and digit set again.

So there's an image with the name tnbanner I was thinking I could just use \w within the first set, i.e. [\w\W\d], but that doesn't seem to work.

Is that because \w (or \W\d) only match a single character? So zbanner would return a positive result but tnbanner wouldn't? Well I presumed that's what it was so tried to include a wildcard to broaden the range.

/[\W\d]*banner(s|id\=)[\W\d]/ or /[\W\d*]banner(s|id\=)[\W\d]/ doesn't work either. Urgh RegExp is baffling, don't have the kind of logical mind to do this kind of stuff.

[rue]

Melon:
banner(s|id\=) - matches only "banners" and "bannerid="
.
The string you're testing -- "tnbanner" -- doesn't end in either "s" or "id=". It would be easy to zero-multiply that final OR-set ...except the whole point was to prevent false-positives on "banner". Instead, just add your unique term as a conditional:
/[\W\d](tnbanner|banner(s|id\=))[\W\d]/

[Melon]

Ahh so it doesn't match banner on it's own then? Interesting.

Oh and yeah, I was thining of adding the tnbanner as a conditional, but that was just an example. Guess I'll stick with my original wildcard happy version.

Cheers for the help.

[Org]

[Melon]

Right I see, I was under the initial impression that the filter encapsulated things like banners, bannerid, banner and also go96banner3f.jpg for example.

But upon closer inspection, I must have totally missed the whole point of the string.

Thanks for the explaination Org.

[Guest]

i think my head will explode....lol

[Org]

Regular expressions may seem a bit hard at first, but actually their basic use is easy to learn. If there is any starting difficulty, it comes from their great power and flexibility. Luckily plenty of help and tutorials are available. Take a look at these:

http://www.icewarp.com/Knowledgebase/402.htm

http://it.metr.ou.edu/regex/

http://java.sun.com/docs/books/tutorial/extra/regex/

http://www.regenechsen.de/regex_en/regex_1_en.html

http://www.amk.ca/python/howto/regex/

http://www.gnu.org/software/emacs/manual/html_chapter/emacs_14.html#SEC97

http://www.regular-expressions.info/

Remember that there are a lot of slightly different rexexp dialects out there. You can learn the basic use from any of these tutorials and help pages, but for exact syntax you should check the JavaScript documentation:

http://devedge.netscape.com/library/manuals/2000/javascript/1.5/guide/regexp.html#1010689

Learning the basics of regexps is more than worth the trouble. And Adblock is not alone, there are lots of programs and tools that use regexps.

[Guest]

thanks for those thinks...

[Guest]

teaches me to spellcheck, i meant LINKS...hehe

[Bethrezen]

Hi

Ok well after that totally confusing basic explanation I think its time I had a go my self coz I'm not going to get anywhere other wise coz this is all totally confusing gibberish to me

See from my point of view you all seem to work the opposite of the way I do see what I tend to do is to block each of the bits that I want blocked then I look at the list of blocked bits looking for some patter so that I can come up with a filter that will block the required bits so that I don't need a big list

see now my first though would be to use

http://www.wilderssecurity.com/YaBBImages

However this is to general and blocks things I don't want it to and this is where I run in to trouble because I have no idea how to get it to block just the listed items

Maybe ill understand a bit better once I'm able to formulate a few of my own filters

Anyway back to the tutorial

Ok ill use wilders as an example again here is a list of images I have blocked on the main page

http://www.wilderssecurity.com/YaBBImages/wilders.jpg
http://www.wilderssecurity.com/YaBBImages/off.gif
http://www.wilderssecurity.com/YaBBImages/open.gif
http://www.wilderssecurity.com/YaBBImages/xx.gif
http://www.wilderssecurity.com/YaBBImages/guest.gif
http://www.wilderssecurity.com/YaBBImages/info.gif
http://www.wilderssecurity.com/YaBBImages/online.gif
http://www.wilderssecurity.com/YaBBImages/php.gif
http://www.wilderssecurity.com/YaBBImages/mysql.gif
http://www.wilderssecurity.com/YaBBImages/vhtml.gif
http://www.wilderssecurity.com/YaBBImages/vcss.gif

Now what I'm wondering is how do I formulate a filter to block all them so that I don't have to have each blocked individually which is how I have it right now coz I got no idea what I'm doing

[rue]

Bethrezen:
/wilderssecurity\.com\/yabbimages\/(wilders|off|open|xx|guest|info|online|php|mysql|vhtml|vcss)\.gif/
.
And, here's the breakdown of that pattern:

/wilderssecurity\.com\/yabbimages\/(wilders|off|open|xx|guest|info|online|php|mysql|vhtml|vcss)\.gif/
/wilderssecurity\.com\/yabbimages\/(wilders|off|open|xx|guest|info|online|php|mysql|vhtml|vcss)\.gif/
/wilderssecurity\.com\/yabbimages\/(wilders|off|open|xx|guest|info|online|php|mysql|vhtml|vcss)\.gif/
/wilderssecurity\.com\/yabbimages\/(wilders|off|open|xx|guest|info|online|php|mysql|vhtml|vcss)\.gif/
/wilderssecurity\.com\/yabbimages\/(wilders|off|open|xx|guest|info|online|php|mysql|vhtml|vcss)\.gif/

The vertical-line is the "OR" conditional. It means any one of the items in parenthesis can match. Forward-slashes (/) and periods (.) have special meanings, just like the OR-conditional (|). To use them as regular characters in the pattern, they must be preceeded with a back-slash (\) -- that's why you see "\/" and "\." in the pattern.
.
Regarding the OR-conditional and its enclosing parenthesis, see pt.ii of the tutorial.

[Guest]

believe or not i'm beginning to comprehend.....my exploding head is slowly rematerializing.....


Many thanks

[Bethrezen]

????? err ok im really not geting this that just looks like alot of nonsence to me ?????

i know you are doin ya best to try and help me out but im geting nowhere i havnt got clue could this be any tougher to understand *sigh* there has to be a simpler way to aproch this coz this is compleatley over my head and as hard as i try im geting nowhere

[rue]

Bethrezen:
Sorry- we did our best :P

[MW]

i found this page helpful

http://www.jamesrking.com/left_brain/reference_repository/regular_expressions.asp

[MW]

perhaps this thread should be sticky and topic changed to "A Quick Giude to REGEXP (regular Expressions)"

[rue]

MW:
The "helpful threads" sticky links here.
.
Multiple stickies made it difficult to locate new posts at-a-glance.

[kp]

I've read most of the replies and as a programmer myself find that the average user will have problems with regular expressions. Although they are powerful they present a rather large learning curve for the 'average user'.

As Mozilla Firebird is aiming for the end user which accomodates the 'average user' is it not possible at least to have some sort of wizard that helps the user to build the regular expression? and/or have a list of basic filters (which I see is the discussion in another topic) ship with adblock.

I do find adblock very useful but something like this may hinder it's take-up.

[kstahl]

Well, an average user could always add filters the "traditional way.

*/ad.*
*/ads/*
*/banner/*
*.doubleclick.net
etc.

(The wildcards are not needed any longer I think?)

That's what I did. When I started with adblock I had no idea how to use regular expressions.

[rue]

kp:
some sort of wizard that helps the user to build the regular expression
You mean like... this?
.
(discovered in a bannerblind bug, oddly enough.)

[kp]

[kp]

Just a note on my post above:

Although the option chosen by rue is good, we must consider the fact that:

1. Having "string" in the menu's text isn't a good idea and

2. Maybe to build a good regular expression takes many menus which could clutter the screen and make it even more confusing for the user.

[time2read]

have you seen the RegEx Coach?

http://www.weitz.de/regex-coach/

[zero ant]

I'm sorry for bringing this old topic up, but I just want to clarify (and fix) something.

----------------------------------------------------------------------------------------------------

[guest]

I'm not allowed entry to devedge.netscape.com. What's up?? I need the syntax info.

Remember that there are a lot of slightly different rexexp dialects out there. You can learn the basic use from any of these tutorials and help pages, but for exact syntax you should check the JavaScript documentation:

http://devedge.netscape.com/library/manuals/2000/javascript/1.5/guide/regexp.html#1010689

Learning the basics of regexps is more than worth the trouble. And Adblock is not alone, there are lots of programs and tools that use regexps.[/quote]

[kstahl]

Unfortunately, Netscape took Devedge offline. I think efforts are underway by Mozilla to save the contents.
_________________
Adblock 0.5.3.042
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1. Gecko/20051111 Firefox/1.5

[wonkothesane]

I cached the JS RE syntax page at http://eschew.org/projects/adblock/jsregexp.html

It's very much a temporary solution until Mitchell gets a solution wrangled.

[Zachariah]

[cell1527]

I found a good site with regular expressions and examples, if anyone is interested:

http://gnosis.cx/publish/programming/regular_expressions.html

[Guest]

[wonkothesane]

For now, try http://devedge-temp.mozilla.org/library/manuals/2000/javascript/1.5/reference/regexp.html#1193136

[RejZoR]

Ok,i'm still examing syntaxes (hard stuff for now ).

All i need for now is to exclude specific URL from already used AdBlock filter (since there is no whitelist option).

I'm using */affiliates/* and it's working great. Problem is that this one also blocks Firefox logos/buttons from: http://www.spreadfirefox.com/?q=affiliates/homepage

Now what do i have to do to exclude:

http://sfx-images.mozilla.org/affiliates/*

from this filter:

*/affiliates/*

How do i do this?

Many thanks!

[Henk]

Rue and cardinal,

Keep up the good work and especialy your endless patience!

As with anything, regex are complex until understood.
Thank you for your clear explaination, for you made it simple for me.

Thanks again!

[jefree]

Hallo developers (rue, cardinal, and the others )

Would you guys mind to update link in adblock help - RegExp Syntax
to

[mcm_ham]

Actually the proper new site is:
http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Reference:Objects:RegExp

Right now a collection of fixes is being made available here:
http://aasted.org/adblock/viewtopic.php?t=2264